1 of 90

Crossing the chasm:

Amplifying success stories about co-creating across institutions

Eli Holmes, NOAA Fisheries, NMFS Openscapes

Julie Lowndes, Openscapes

Jen Schopf, Texas Advanced Computing Center, UT Austin

Amy Steiker, NASA National Snow & Ice Data Center �(NSIDC DAAC), NASA Openscapes

Yuvi, 2i2c / Project Jupyter

And many other contributors!

Slides: https://openscapes.org/media

ESIP Meeting, July 2025

Why are we here

Since our keynote at CNG conf, we’ve been retelling across NOAA
Power of stories, of sharing successes. Prioritizing, pushing ourselves. Real examples to amplify & fork – just as you do code.
4 stories of crossing the chasm
Plenary-style discussion
Collaborative notes doc

The RDEC focuses on strengthening RDEC: NOAA’s R&D enterprise through effective planning, monitoring, evaluation, and reporting.

Julie: Hello and thank you, happy to be here

Eli: Hello and thank you

Julie: power of amplifying and forking stories as collective action. Have the privilege of sharing this work that is the effort of a lot of people.

Short intro/background:

Eli Holmes (NOAA Fisheries) and Julie Lowndes (Openscapes) will present “Crossing the Chasm: Helping Organizations Innovate Through Workforce Modernization.” Since 2021, they have led a major effort to modernize the scientific workforce at NOAA Fisheries, working with nearly 600 staff across disciplines. Their work focuses on building capacity for reproducible, efficient, and transparent science—through hands-on support and training to streamline data access and analytical workflows, migrate from manual to automated/scripted data workflows and improved documentation. By fostering open collaboration, reducing duplication, updating programming skills for new data and tools, addressing infrastructure and policy barriers, this initiative has enabled staff to improve and modernize their data workflows together, as well as organize and plan field campaigns and cruises more effectively.

2 of 90

Diffusion of Innovation Theory & Crossing the Chasm

Eli Holmes

NOAA Fisheries

3 of 90

About us:

We have a long history of actionable science & teaching

Julia Lowndes, PhD

Openscapes Lead

15+ years marine ecologist

Eli Holmes, PhD

NOAA Fisheries Open Science Lead

25+ years at NOAA Fisheries as a statistician/NMFS PARR rep

Why are we here today?

Share what workforce modernization can look like & how it improves our science
Let you know what is happening now (and since 2021) at NOAA Fisheries

We’re representing the work of many people across NOAA Fisheries and beyond!

4 of 90

Chasm between your great idea and wider adoption

Early Adopters

5 of 90

Early Adopters

Sharing a success stories

Target Population

Crossing the chasm is really hard. Sharing successes helps keep us moving forward.

6 of 90

Why is the chasm such a common phenomena?

A sheepdog parable…

Hand off to Eli

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

7 of 90

“Crossing the Chasm” – selling a sheepdog

8 of 90

The idea: initial inspiration

The innovator watches wolves hunting and has a crazy out-of-the-box idea: What if I could take a wolf and train it to use this behavior to herd sheep!

9 of 90

Problem is, the innovator is not really suited to turning this vision into reality.

The innovator is fascinated by the wolf and is not bothered by the complete impracticality of the wolf.

We need something considerably more tractable and practical.

10 of 90

Early adopters - take a wild idea, relentless drive to turn it into a workable product.

Take a puppy

Patience and skill, actually obsession, to do this

11 of 90

The early adopter

The target population

Then it is time to ‘sell’ the idea to those who raise sheep (the target audience)

The early adopter can’t help trying to get the sheep raiser to love puppies “Ok, so it’s a puppy and needs training!! JUST LEARN TO TRAIN THE PUPPY LIKE I DID! It’s FUN! *

I raise SHEEP! I don’t care about puppies*. I don’t want to train puppies**. I want to raise sheep.

*The early adopter dies a little at this statement…. **depression sets in

12 of 90

Social & operational chasm

Institutional barriers

Early Adopters

13 of 90

Crossing the Chasm at NOAA Fisheries

Eli Holmes

NOAA Fisheries

14 of 90

Early Adopters

How do you build the bridge to cross the chasm?

Target Population

Adoption of Reproducible Science and Data Workflows� at NOAA Fisheries

15 of 90

Where we are today

Before we get into this, let’s show you where we we are today. Then we will talk about how we approached getting here.

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

16 of 90

Community Building and Skill Building

Openscapes

Support for ‘mentors’, organizers, leads

Data Science academy

1:1 and Team�Support

Infrastructure

training Jupyter Hub

Weekly Trainings and Hack Events

Training Page

GitHub
Quarto Reports
R
Python
Cloud Computing
Remote Sensing Data

Google Spaces for Posit Connect, GitHub, R, Python, Open Science

AMA Help Desk with Jon Peake every Wednesday.

asar team also is holding weekly helpdesk

Training Page has sign-up links

Team trainings. Email Jon and Eli.

Weekly Coworking

Fall Openscapes Champions cohorts

Help for on-boarding to Cloud Computing

Posit Connect server for Shiny applications and data dashboards

GitHub Enterprise server for code management & preservation. Governance Team

2 FT staff

E Holmes & J Peake

“Mentor cohort at each FMC”

2025

J Lowndes, S Butland,

A Teucher, I Fenwick

17 of 90

How did we get to today?

Coming up with a strategy using Diffusion of Innovation theory

in 2021

Approached as organizational change through design thinking.

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

18 of 90

“Diffusion of Innovation” Theory (EM Rogers 1962)

Predictable progression as an idea diffuses through a population.

Important to understand so you don’t put your energy in the wrong place.

Time

I have a few talks on this idea on my website:

eeholmes.github.io

Target Population

19 of 90

Early Adopters are critical to diffusion of innovation

Early Adopters develop the innovation into something of value
Their energy and effort is what drives the initial diffusion process, but that is a slow process.

Time

Target Population

20 of 90

2020-2021

2024-2026

2022-2023

First 4 Champions Cohorts

Roll out of Agency-wide training program

Developing the Mentor Cohort + co-leading 6 Champions cohorts

Identifying is important, but how do you execute the vision. Start small, keep building. We made this slide in 2021. It helped keep us on track of what to do at each stage.

What does it look like for me, Julie. I’m helping this organization. This was the planning, a lot of Eli’s thinking about this. I’m helping execute as the facilitation, in my role as I help organizations. What did this look like? Echos the process that Openscapes has done with other orgs: Fred Hutch, NASA, Water Boards. How to creat something self sustaining

Different audiences, growing examples. Say what’s hard out loud. Balance vision with specificity and details.

Vision

Across science centers and regional offices
Train a generation of NMFS project leaders in Open Science, inclusive team culture, and modern project management

21 of 90

How do we get started? People don’t want to try new ideas – in fact they kind of actively resistant to our new ideas.

Start with the eager

and then move to the willing.

Approached as organizational change through design thinking.

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

22 of 90

You need to find the “eager” (the Early Adopters), but they are hard to reach and many are isolated

Early Adopter

23 of 90

Developing and growing the Early Adopter Community at NOAA Fisheries

2021-2023

Find Allies

Help people/orgs/teams with their “pain points” and goals

Contribute

Community

24 of 90

Start with the eager and then move to the willing.

Remove barriers to joining your group.

Cultivate culture of openness and helpfulness.

You have to do 10x more comms than you think.

Approached as organizational change through design thinking.

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

25 of 90

Early Adopters

What’s your success story?

Target Population

26 of 90

Crossing the Chasm at NASA Earthdata, 2021-2022

Julie Lowndes

Openscapes

27 of 90

Date: Tuesday, July 29, 2025 << please join us!!�Time: 9:30 - 11:00 am PT�Speakers: Amy Steiker, Luis Lopez, Danny Kaufman, Joe Kennedy, Chris Battisto

Register (free) via Zoom:

https://openscapes.org/events/2025-07-29- community-call-earthaccess

A python library that enables authentication, search, & access for NASA Earth science data with just a few lines of code.

28 of 90

NASA Earthdata & Openscapes

Supporting NASA Earth science research teams’ migration to the cloud

https://nasa-openscapes.github.io

The overarching vision is to support scientific researcher teams using NASA EOSDIS data as they migrate their workflows to the cloud. We are doing this working with NASA Distributed Active Archive Centers (DAACs) over three years by:

Develop a cross-DAAC Mentor community of collaborative cloud data instructors, that co-create, curate, and use shared resources (“make once, use often”)
Empower science teams through the Champions program to migrate their download- intensive data analysis workflows to the cloud and open, kinder science
Scale the Openscapes Champions program with DAAC Mentors to support more teams transforming their workflows towards open, kinder science and the cloud

Slide from AGU, Robinson & Lowndes 2021

29 of 90

EarthData Cloud Cookbook

In active, modular, open development

nasa-openscapes.github.io/earthdata-cloud-cookbook

Create place:

GitHub Org + Quarto

Purposeful documentation

Create-once-use-often, importer workflow
Researcher-focused: tutorials & how-to guides

quarto.org

Common

Cloud

Foundation

NASA Specific Science Tutorials

Slide from AGU, Robinson & Lowndes 2021

30 of 90

Mentors’ early co-creation and teaching was key

* four other workshops

Time

Target Population

Nov 2021:

Cloud Hackathon

Mar 2021:

Kickoff

Oct 2022:

Luis Lopez demos earthaccess

(then called earthdata)

Apr-Jun 2022:

Champions Cohort

Designed as part of their paid jobs, ~20%

*

31 of 90

Building the mechanism:

Openscapes Flywheel

(Robinson & Lowndes 2022)

(an open source tool to fork & reuse!)

32 of 90

The Openscapes Flywheel: an open source tool to facilitate and scale inclusive Open science practices

Robinson & Lowndes 2022

Fork common workflows, skills, tools, ideas

Work Openly

Invest in learning and trust

Inspire

Flywheel

Create space and place

Welcome

Empower

Learning culture

Engage

A Future Us mindset

Amplify

Open leaders

Flywheel concept

transformations occur from consistently doing key activities that add up over time

(Collins)

33 of 90

Coworking

Regularly scheduled, with people not part of your normal work team

Cross-team awareness; cross-org learning

Solve problems; turn around and share

Onboarding

Work Openly

Invest in learning and trust

Inspire

Flywheel

Create space and place

Welcome

Forking as a worldview

I want to talk about coworking bc this is one of the most actionable, low cost, immediate things groups can do. That doesn’t mean that it is easy or just happens: it’s work by somebody, which is why it’s not free. setting the intention and the vibe of coworking really matters, it’s real intention and strategy.

What is coworking? It comes in different flavors. It is about getting your own work done, and joyful accountability.

Coworking is where someone leans forward in their chair while a colleague is screen sharing and says “I need that. Can you teach me?”

Cross-team awareness: help identify common challenges and solutions, grow a coalition to roll out further

We see the trust and expectations established in synchronous coworking carry over into asynchronous communication spaces like Slack or Microsoft Teams and in in-person and hybrid interactions.

34 of 90

“It was a really great week. The tutorials were AMAZING. Everyone did a great job, and everyone was very nice. I really appreciated welcoming environment. I don't have a strong python background. But i was supported in learning all around”

Blog summaries:

earthdata.nasa.gov/learn/articles/2021-cloud-hackathon

podaac.jpl.nasa.gov/announcements/2021-12-15-The-2021-Cloud-Hackathon

65 Openscapes 2i2c JupyterHub AWS instances

50 forks of the Cloud Hackathon GitHub repo

8 hack-team projects presented on Day 5

Empower Research Teams

2021 Cloud Hackathon

November 15-19

https://nasa-openscapes.github.io/2021-Cloud-Hackathon/

Slide from AGU, Robinson & Lowndes 2021

35 of 90

Hackathons

Tutorials + peer-to-peer learning + project team-based work (eScience definition, hackweeks)

A lot of generative ideas – identifying a problem so painful

Invest in learning and trust

Work Openly

Inspire

Flywheel

Welcome

Forking as a worldview

Create space and place

I want to talk about coworking bc this is one of the most actionable, low cost, immediate things groups can do. That doesn’t mean that it is easy or just happens: it’s work by somebody, which is why it’s not free. setting the intention and the vibe of coworking really matters, it’s real intention and strategy.

What is coworking? It comes in different flavors. It is about getting your own work done, and joyful accountability.

Coworking is where someone leans forward in their chair while a colleague is screen sharing and says “I need that. Can you teach me?”

Cross-team awareness: help identify common challenges and solutions, grow a coalition to roll out further

We see the trust and expectations established in synchronous coworking carry over into asynchronous communication spaces like Slack or Microsoft Teams and in in-person and hybrid interactions.

36 of 90

Previously unseen barrier

Core functionality for data access – Earthdata Login authentication, efficient search and query via CMR, bulk download, and direct access from AWS object stores – was insurmountable for many users via the code-based workflows required for Cloud.

https://nasa-openscapes.github.io/

2021-Cloud-Hackathon/tutorials/

04_NASA_Earthdata_Authentication

NASA’s goal to support Earthdata users migrating to Cloud workflows would not have been as successful without the earthaccess Python library. earthaccess solves a large and previously unseen barrier: that core functionality for data access – Earthdata Login authentication, efficient search and query via CMR, bulk download, and direct access from AWS object stores – was insurmountable for many users via the code-based workflows required for Cloud. earthaccess provides a unified approach to reproducible data search and access, whether users are working on a local machine or in the cloud, and regardless of where the data itself is stored (Figure 1, left). earthaccess is widely and increasingly used – 200 dependencies; 100+ dependent projects – and because earthaccess is integral to Cloud adoption, earthaccess is a success story tracking an increase in NASA Earthdata Cloud users.

37 of 90

Partnering

Early adopters can’t do it alone. Must partner, bring new creativity & resources

Many forms of contributors:

Advocating for time

2i2c; RStudio/Posit

Cheatsheets, Art

Work Openly

Invest in learning and trust

Inspire

Flywheel

Create space and place

Welcome

Fork common workflows, skills, tools, ideas

I want to talk about coworking bc this is one of the most actionable, low cost, immediate things groups can do. That doesn’t mean that it is easy or just happens: it’s work by somebody, which is why it’s not free. setting the intention and the vibe of coworking really matters, it’s real intention and strategy.

What is coworking? It comes in different flavors. It is about getting your own work done, and joyful accountability.

Coworking is where someone leans forward in their chair while a colleague is screen sharing and says “I need that. Can you teach me?”

Cross-team awareness: help identify common challenges and solutions, grow a coalition to roll out further

We see the trust and expectations established in synchronous coworking carry over into asynchronous communication spaces like Slack or Microsoft Teams and in in-person and hybrid interactions.

38 of 90

Luis Lopez presenting then-called earthdata Oct 2022!�

39 of 90

Co-creation

Hackathons

Teaching

Coworking

YES!

Reusing what works in new places

40 of 90

Creating our own certainty

Time to learn & contribute
Seeing the problem so painful
Partnering - asking for help

Hand off to Eli

Organizational change through design thinking. Trying to understand your user team. You don’t want the product as much as you want them to help teach you how to make cool products

Success is not clear. Things we can do. We want to share with you how to increase success.

Many “small” things, not one big thing. It’s not lone hero, it’s collaboration

Build mechanisms & skills across the community. Not products only; process. Teach me how

Data academy: worked really hard to fill 40 spots. Have 150 spots.

(specifics) Mentors from all 6 science centers,

Champions: fall we mobilized after a year, developed a plan and turned around to start 3x Champions with 2 months notice

Data academy: expected to work really hard to fill 40 spots. Had 150 applications.

3 years of open development to get us here.

We had a plan, a vision. Xyz steps. But it has been more than that. And meeting this moment.

Current picture: Product is the thing. Reject: it’s the process, changing the whole org to do something different. An organization where people feel they have more agency

Eli was introduced to a theory that we used as a roadmap -

41 of 90

Date: Tuesday, July 29, 2025 << please join us!!�Time: 9:30 - 11:00 am PT�Speakers: Amy Steiker, Luis Lopez, Danny Kaufman,

Joe Kennedy, Chris Battisto

Register (free) via Zoom:

https://openscapes.org/events/2025-07-29-community-call-earthaccess

42 of 90

Crossing the Chasm at NASA Earthdata

Amy Steiker & Julie Lowndes

NASA NSIDC DAAC & Openscapes

To be presented Tuesday, July 29, 2025 << please join us!!�Time: 9:30 - 11:00 am PT�Register (free) via Zoom –

https://nasa-openscapes.github.io/news/2025-07-29-community-call-earthaccess/

43 of 90

Time

Hard to even define the chasm

It’s hard to even define a chasm.

Ian thoughts - 2 chasms: here’s where we thought we were and were going, but…

People using earthdata cloud (earthaccess tool is the beginning of the infra to cross)

2) small community of people developing earthaccess, started with a bit of funding. But now need to fund independently of esdis and grow – wait this is what’s hard about bridge building,, not a chasm

Amy’s thoughts: If we’re talking going from idea to adoption - solving barriers to access, especially in the context of Earthdata Cloud, was a huge chasm - imo, based on organizational structure and narrow lens of “value streams” (aka the box was checked if a file was “distributed” successfully, and UI tools were successfully built, but not considering the primary programmatic workflows of our users). That chasm was solved through Openscapes - bringing multiple perspectives and backgrounds together, breaking down organizational barriers. Earthaccess would not have happened without Openscapes… so that was the initial bridge.

44 of 90

Early Adopters

How do we help users leverage the awesomeness of NASA Earthdata?

Target Population

A bridge: earthaccess.

Shift now is in who helps build this bridge

45 of 90

“earthaccess is amazing”

- NASA SMAP Mission PI from Univ. Montana.

earthaccess integral to his open climate science for ag course:

openclimatescience.github.io

We built earthaccess to support users migrating search, access, and analysis workflows to the AWS Cloud.

But it is so much more.

200+ dependent projects¹

¹ https://github.com/nsidc/ earthaccess/network/dependents

500+ GitHub stars²

² https://github.com/nsidc/ earthaccess

This is an ugly start to improve, but thinking this is the “wow” page

AS: Add another box here to highlight hack days? Screenshot from Matt’s recorded demo or hackday post in GH?

Enables reproducibility of workflows, collab & transparency.

Now on earthdata.nasa.gov!!³

³ https://earthdata.nasa.gov/data/tools/earthaccess

46 of 90

We built with intentional culture & community – critical to its success so far & its future growth.

Hackathons & Workshops Cross-learning between developers and users.

Open community

ownership, the ways we support our contributors & maintainers

Community partnerships & collaborations

Xarray • Pangeo • HDF • OPeNDAP •

Openscapes

Amy notes: We’ve faced our own set of challenges as we continue to develop and grow the earthaccess library and community, including budget and funding uncertainties, leaving us void of sustained funding at the present moment.

Although these are no doubt setbacks, what we do know is that this effort to be intentional about building culture and structures around community maintenance and governance, and investing in early adopters has been critical not only to the success of the library, but to NASA Earthdata as a whole.

These aren’t novel concepts - we know that open source/community ownership can accelerate development and reduce cost, but I can’t understate the value of providing these arguably simple ways to implement and enable community support, in order to leverage and take in the existing community momentum and interest/engagement of these early adopters. And these are crucial in a climate where funding is not to be taken for granted and where we must implement creative solutions. We (the big we - that includes our own user communities) can now show up for NASA Earthdata in a very tangible and concrete way - earthaccess as a mechanism for continued community support. Yes we are an innovation engine, we are partnering on some hard problems to solve, but also we are improving very fundamental pathways and workflows for users too.

47 of 90

Early Adopters

Widening the bridge, as the chasm changes

Target Population

Ongoing work: new features & support.

Shift now is in who helps build.

48 of 90

Vision for the Majority: “batteries included”.

Time

49 of 90

Enabling real world analysis at scale: On-demand virtual data cubes

Community governance evolution: “Repotting” earthaccess repo to further enable partnerships and “mature the bazaar”

Continuing to lower barriers to library contributions

What does widening the bridge look like?

50 of 90

Early Adopters

How do you build the bridge?

Target Population

We’ve already built this foundation with earthaccess, we can look at how our community model can support the bigger cause

How to increase partnerships, external funding?

How do *we* support ESDIS, rather than the other way around?

Hitting this chasm allows us to look further; part of a bigger, continued need.

Learning from and furthering stories that already exist

51 of 90

Crossing the Chasm with TACC

Saving 50 years of astronomy data with Arecibo

Jen Schopf

Texas Advanced Computing Center / University of Texas at Austin

52 of 90

WHAT IS TACC

The University of Texas at Austin and UT System Research Computing Facility
Largest NSF-funded national computing center for open science

Also NIST, NASA, NIH, DARPA, DOD, etc.

200+ Dedicated staff
Altogether, ~20k servers, >1M CPU cores, 1k GPUs
About seven billion core hours over several million jobs per year – for 4,000 projects and ~95,000 users per year.

52

5/15/25

Federal Investments in TACC are over $1B in last 10 years; and over $1B slated for next 10 years.

While we are a national provider, we have *by far* the most computing resources of any University

in the country (and often the world), and will continue to through the 2020s.

53 of 90

We help Bridge the CI Chasm for Users-

Provide researchers with:

Computing, Data, AI , Software capabilities to support their research
The expert help to use it
In the ways they want to consume it
Help with grants/strategy

We also like to help Departments and Colleges:

Recruit top faculty and top students.
Win large grants, leveraging our capabilities.

Computation, AI, Data almost ubiquitous across the sciences

53

5/15/25

54 of 90

We have Computers. Lots. All kinds. Especially big ones

From single node to 500,000 processors.
Accelerated computing with GPUs (and a few DPUs, FPGAs, Vector Accelerators, etc.)
Batch scheduled, managed;
Bare metal; Kubernetes/Virtual Machines/Containers
Interactive (Web, API, Jupyter. . .).
Large memory
Simulation, AI/ML/DL, Analytics.
Visualization walls, VR/AR/Displays

54

5/15/25

55 of 90

We have Storage - Lots. All kinds.

A few Petabytes of solid state
A few hundred Petabytes of spinning disk
>Half an Exabyte of tape
Some optimized for speed, some for reliability, some for cost
We have some that you can use for a few days, and some in service since 1986

55

5/15/25

56 of 90

WHO USES TACC

~4,000 active projects, 150+ fields
~15,000 annual command line users
~80,000 annual via web portals, APIs
Partnerships with UT System, NSF ACCESS, industry, international
K-12 programs, college classes, institutes and other training (1,000+ users trained annually)
7 billion compute hours annually
5 billion files processed annually

56

57 of 90

We have begun construction of the Leadership Call Computing Facility (LCCF)
A >$500M new facility with a long term investment in computing, data, people, and facilities for simulation, analytics, and AI.
Funded by the same “Major Facilities” fund as NCAR, LIGO, LHC, Vera Rubin, McMurdo, etc.

This is a promotion for the role of computing

58 of 90

ARECIBO - A HISTORY OF (PLANNED) CHALLENGES

1963 - Arecibo in Operation https://www.naic.edu/ao/legacy-discoveries
2006 - NSF 15% Budget cut across Astronomy
2007 - Arecibo budget cut from$10.5 to $8M

(NASA adds ~$2.6M to help ops budget)

2015 - Facilities director Kerr quits due

to funding clashes

2018 - University of Central Florida

takes on stewardship

58

59 of 90

UNPLANNED CHALLENGES (1)

Hurricane Maria ( Sept 20, 2017 )

Category 4
Significant damage to facility
R&E (I2) network connection lost and not replaced

Too expensive to repair due to budget cuts

Core operations resume when the power came back...

59

60 of 90

UNPLANNED CHALLENGES (2)

5.0 - 6.4 Earthquakes ( January 7-11, 2020 )
Operations still going

shaken not stirred

60

61 of 90

UNPLANNED CHALLENGES (3)

July 30, 2020

Tropical Storm Isaias
SCIENCE vs operations means that some hard choices are made
Maintenance deferrals start adding up

61

62 of 90

UNPLANNED CHALLENGES (4)

Aug 10, 2020

First [auxiliary] cable snap

Nov 6, 2020

Second cable [primary] snaps

2024 NAESM report -

Zinc “creep” slowly deformed the wire support sockets

63 of 90

DECEMBER 1, 2020 - RESULTING FACILITY PROFILE

63

Data Center Building 1.

64 of 90

NEED TO MOVE 2+ PB OF GOLDEN COPY DATA

Option 1: Move data to the cloud

Est time to transfer: 42 years
Est cost: Millions for downloads

Option 2: Move data to partner site at UCF

Partner site says they can’t support this option

Option 3: TACC steps up to save science

Workflow developed
9 months later, all data from spinning disk at TACC

64

65 of 90

MOVING THE DATA

Network Attached Storage (NAS) “Appliances”

100+ Terabytes at a time (Full capacity of NAS)

Onsite team hand carries NAS to the closest 10Gbps links (coastal - IRNC funded AMPATH)

University of Puerto Rico - Mayaguez
Engine-4 Commercial Collaboration Space

Device tuned to be more effective, but still a slow process

65

66 of 90

WORKFLOW TO MOVE DATA ON SPINNING MEDIA

66

67 of 90

DATA AT TACC

~9 months later and 2.5PB of data moved

Arecibo team set up web interface
Maintained on a volunteer basis

Some science projects also made local copies – from TACC, not PR

67

68 of 90

BUT WHAT ABOUT THE TAPES?

68

69 of 90

6,600+ TAPES

69

70 of 90

AO DATA MOVEMENT PART 2

Team from TACC deployed May 2023 to save the Data on Tapes

Approx 60 years of data history was condensed down to 4 pallets
Estimate of about 3.8PB of data (empirically a PB of data weighs 664# …)

Data Currently at TACC

~3 PB Online spinning disk
Rest of Tape data available on demand

Arecibo closed August 2023

70

71 of 90

71

72 of 90

ACKNOWLEDGEMENTS

EPOC was funded by

US NSF award #1826994 through 2023
US NSF award #2328479 through 2025

72

73 of 90

TAKE AWAYS

TACC has resources available to help you bridge your personal data chasm - or collapsing infrastructure…

and interest in helping science, however we can
(although we’d rather not ship pallets of tapes again)

Reach out

Jennifer Schopf, jms@tacc.utexas.edu

73

5/15/25

74 of 90

Finding Chasms to Cross �JupyterHubs at Scale from Wikimedia to UC Berkeley to your Research Institution

Yuvi

2i2c / Project Jupyter

75 of 90

76 of 90

77 of 90

When you grow up in an environment like that, you’re naturally attracted to open source software. You either can’t afford proprietary software, or if it was being offered for free, you knew through shared cultural memory that it’s only being offered for free to extract something valuable from you. You may not know what that it is yet, but you know you’ll pay for it one way or another. The transactional nature of this is absolutely fine, but it’s important to remember that’s what it is.

There absolutely is such a thing as a free lunch - but it doesn’t come from any company, it comes from the community that you’re a part of. It’s a lunch you helped cook, so it can be free for you and for all others.

I started contributing to open source software from a young age, and eventually ended up in the operations team for Wikimedia, the org that runs Wikipedia. Imagine Wikipedia, serving billions of people, run by an operations team of less than 15! I was the youngest and least experienced person there, but I learnt a lot. I picked up community building skills, technical skills, time management skills and a ton of other things. It wasn’t without its problems, but the lunch was hearty. Thanks to Sumana Harihareshwara, Ori Livneh, Faidon, Bryan Davis, Ryan Lane and many many others for giving me tools and skills!

78 of 90

Fast forward to 2015, and I’m a bit restless. I felt the desire for some more fulfillment in my life. And through sheer luck, I ran into the data8 project at UC Berkeley. It had some absolutely amazing people trying to solve a specific problem - how do we systemically support the thousands of students without any prior programming experience who now wanted to learn datascience and move forward in their careers?

To serve students who didn’t have prior programming experience, it was really important that they be able to jump into executing code as soon as possible, without having to go through lab setup or installing python or learning commandline tools via ssh or anything like that. You couldn’t serve a large number of students that way - it was pedagogically the wrong thing to do.

They had amazing leaders (Cathryn Carson), pedagogically gifted instructors (John Denero, Ani Adhikari and David Wagner) and wonderful infrastructure people (Ryan Lovett). Here was a chasm to cross, and a bunch of people trying to cross it.

79 of 90

English Wikipedia

296,000,000,000 pageviews in 2024

Data8

~200 �users in Fall 2015

80 of 90

200 -> 1500 students within 2 years

81 of 90

Crossing Chasms always requires a multitude of skills

82 of 90

Extract z2jh out, so it can serve a broader community

83 of 90

Build community around z2jh

84 of 90

Becomes a tool others can use to cross their own chasms!

85 of 90

2i2c.org, a cloudy non-profit to contribute cloud-y skills to many chasm crossings

86 of 90

You have skills and tools that you have collected through your unique life path

You may not know what all you have!

87 of 90

Crossing Chasms always requires a multitude of skills

88 of 90

Find people whose life paths didn’t look like yours to cross chasms together with

89 of 90

A group somewhere may be missing exactly just your skillset to cross a chasm

Explore the unfamiliar

90 of 90

Together we can go where none of us could go alone