Stat 159/259

Collaborative and Reproducible Data Science

​

Fernando Pérez

Jimmy Butler

Sequoia Andrade

Your awesome TAs

A bit about me

• Physics undergrad in Colombia:
• Open science (Linux, ArXiV, …).
• Particle Physics (QCD) in Colorado:
• Software "on the side"
• A long journey through applied math, neuroscience, data science…
• Woven with open source software
• Today faculty @ Berkeley, LBNL, direct several initiatives where open source tools are essential.

IPython and Jupyter - a lesson in the power of open communities

IPython: "an afternoon hack", Oct 2001

Notebooks: Computational Narratives

• Interactive exploration is...
• a conversation with the computer,
• and with humans,
• woven into a shareable, reproducible narrative.

10.1109/MCSE.2021.3059263

Image credit: Scriberia, for the Turing Way Book Dashes.

Jupyter is an open community dedicated to�modular, platform-agnostic tools for interactive computing

Scientific Software: more than code

Content and Services (Notebooks)

Software

Standards and Protocols

Community

Content & Services

earthcube.org/notebooks

Explosive Adoption

github.com/earthcube2022/ec22_markowsky_etal

github.blog/news-insights/octoverse/octoverse-2024

The Jupyter architecture in a nutshell

Taken from our IPython in Depth tutorial, where you can find a lot more details.

A language agnostic protocol

~100 different kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Community: formalized governance

Large, inclusive, multi-stakeholder community. jupyter.org/governance

Governance structure

• Elected Executive Council (6 people)
• Software Steering Council (large)
• Jupyter Foundation Governance Board (external organizations - funding and partnerships)

​

Community recognition

• Distinguished Contributors

2024 Executive Council Members

Community: where all the credit goes

And thousands more open source contributors!

Scientific Python: a whole ecosystem

More than code, woven into science

Services and content: impact

Software

Standards and Protocols: ecosystem

Community: innovation & resiliency

People

Ideas

Tools

Stories

Tools for the Life-Cycle of research ideas

• Exploring:
• Collaborating:
• Production:
• Sharing:
• Teaching:

Markdown, code, computation…

Data

Community-extensible: NeuroHackademy 2018

Credit:

​

Anisha Keshavan

Nate Vack

Chris Gorgolewski

Real-Time Collaboration!

​

​

Notebooks, text files, …

​

​

Formatting this video so it looks great on any device. Check back later to view it.

Jupyter-AI: vendor agnostic, fully open…

blog.jupyter.org/generative-ai-in-jupyter-3f7174824862

Production: cloud or HPC!

Harnessing the power of cloud computing to study the whole Earth interactively

Interactivity

Distributed computing

Data models / numerics

https://mystmd.org

1. Markup Language
2. Specification
3. Open Source Tools

Slides credit:

Rowan Cockett (CurveNote)

The future of publishing/sharing

New publication workflows…

Impact:

science, education, industry, …

Black holes: LIGO, Sept 14, 2015

2019: More Black Holes! Swiss Open Science Day @ EPFL

Katie Bouman

Caltech

Data8:

Computational and Inferential Thinking

The focus of the course is on reasoning, visualization, and interpretation, rather than calculations or the use of software packages. This approach is inspired by the boldly innovative Statistics by Freedman et al. (1978), a textbook that transformed the way the field of statistics was introduced to undergraduates at Berkeley and around the world.

[...]

However, the presentation of these classical lessons changed sharply when we adopted computation as a central tool

[...]

When students open up a browser-based Jupyter Notebook in a data science class, that action is as natural to them as breathing, regardless of their background or academic interests.

hdsr.mitpress.mit.edu/pub/e69066t4/release/3

Data Science Education at Scale

Data 100:

~1,300

Data 8:

~2,000 students

data.berkeley.edu/2023workshop

​

• Non-profit spinoff
• (Berkeley, Columbia, UBC)
• Business model: service provider for interactive cloud infrastructure.
• Serves research and education, supports Open Source.

Funding

(Open Science Program)

1. Fortran (1957)
2. FFT (1965)
3. Biological databases (1965)
4. Global Circulation Model (1969)
5. BLAS - Numerical Linear Algebra (1979)
6. NIH Image (1987)
7. BLAST - Gene sequencing (1990)
8. arXiv.org - Preprints (1991)
9. IPython/Jupyter notebook (2011)
10. AlexNet - Deep Learning (2012)

https://www.nature.com/articles/d41586-021-00075-2

Credit: Maryam Zaringhalam, NIH/NLM/OSTP

Embraced by Industry (large & small!)

OpenAI card for o1 model

Security evaluation, p. 17.

With this open architecture,

we are building in new spaces!

Agile Metabolic Health & JupyterHealth

Jupyter Health architecture: research and operations

SMART-on-FHIR visualization app

Jupyter Health

Runtime �AI/ML platform

FHIR server

“Dev”

“Ops”

Actual workflow with my data

Jupyter, in the cloud, with my BP data

jupyter-health.2i2c.cloud

Collaborative, cloud-native 3d modeling

GeoJupyter

​

Open Platform for Earth/Geospatial Science

GeoSpatial data: the "crude oil" of modern data.

• Science
• Climate
• Ecology
• Remote sensing
• Engineering
• …
• Government & policy
• Urban planners
• Transportation managers
• Emergency response agencies
• …
• Industry
• Agriculture.
• Finance
• Logistics
• …

Image: GPT-4o

My concern: proprietary capture!

US hazards data is locked behind proprietary ESRI tools, b/c FEMA integrated their Hazards Model for Natural Catastrophes into the ESRI ecosystem.

JupyterGIS - rapid prototyping

Try it in your browser! github.com/geojupyter/jupytergis

2023

NASA’s Year of

Open Science

49

NASA Transform to Open Science Mission

Dr. Chelle Gentemann, Science Lead

Yvonne Ivey, Equity Lead

Cyndi Hall, Community Coordinator

Isabella Martinez, Content Coordinator

Dr. Yaitza Luna-Cruz, TOPS Program Officer

Dr. Paige Martin, TOPS Program Officer

​

Kevin Murphy, Chief Science Data Officer SMD

Katie Baynes, Deputy Chief Science Data Officer SMD

Dr. Steve Crawford, Science Policy Officer SMD

Andy Mitchell,

Dr. Elena Steponaitis, SMD Development Program Executive

Amy (Uyen) Truong, Chief Science Data Office Coordinator

Dr. Rachel Paseka, OSSI Program Officer

Dr. J.L. Galache, OSSI Program Officer

Dr. Demitri Muna, OSSI Program Officer

Molly Adams, OSSI Coordinator

​

TOPS Email List

TOPS Website

SCIENCE MISSION DIRECTORATE

Open-source science at the forefront

github.com/learnopenscience

50

ChelleGentemann

A quick reflection on the explosive growth of the AI/ML scene

​

​

(obviously the $$$ of industry investment helped a little 😉)

A culture of rapid, open iteration

Code

Data

Publications

Platforms

"Open Source is eating our lunch"

Leaked internal Google Document, May 4, 2023

​

semianalysis.com/p/google-we-have-no-moat-and-neither

GitHub: Python, Jupyter and AI

Jupyter-AI: vendor agnostic, fully open…

blog.jupyter.org/generative-ai-in-jupyter-3f7174824862

STAT 159/259: Collaborative and Reproducible Data Science

• Why?
• An essential concern of modern computational research.
• Social and scientific implications of lack of reproducibility.
• Frame the problem in terms practical, ethical and epistemological.
• What?
• Core ideas: data access, computation, statistical analysis and publication.
• How?
• Skills and habits necessary to make a practice of reproducibility.
• An everyday practice, not a “publication time” concept.

STAT 159/259: Collaborative and Reproducible Data Science

• Schedule: 3h Lectures, 2h lab
• Prerequisites: foundations in computation, probability and statistical modeling
• Enrollment: ~ 50 undergrads, 10-20 grads, multiple majors.
• Grading: weekly readings, quizzes, homework and 3 projects.

bit.ly/stat159-f17, bit.ly/stat159-sp21, bit.ly/stat159-sp22

Philip Stark

Statistics ('21)

Eli Ben-Michael

Statistics (‘17)

Daniel McAndrews

iSchool (‘21)

Facu Sapienza

Statistics (‘22)

Facu is at AGU in person. Talk to him!

Why? Ideas: readings

• A few papers/blog posts/videos each week.
• Reflections on computing, reproducible research and open science.
• Tackle
• epistemological questions on scientific validity
• challenges in reproducibility
• changes in modern scientific practices
• questions about the scientific community, incentives, …
• Basis for weekly ~½ h discussion during lecture.

What/How?

Practical skills, underlying concepts

Computational hygiene: a daily habit

GitHub Classroom for everything

Explicit dependency management

Open publishing with Jupyter Book

Open interactive textbooks with Jupyter Notebooks

How can we make the notebook a publishable document?

mybinder.org: shareable reproducibility

Explicit Dependencies

+

+

Continuous Integration with GH Actions

We use a (standard, canned) workflow, and simply edit, commit and push…

GH Actions automatically does the rest: run our deploy-book workflow, which triggers a bot's GH Pages website deployment.

A theme for the course: earth & climate

• A unifying thread: open science, reproducibility, science in the cloud
• More than ML for industry
• Beyond CSVs / Data Frames
• First-principles models meet data-driven methods

Homework: Real-World Reproducibility

Final Project: full research compendium

• Main narrative notebook ("the paper"): summarizes and discusses results.
• All analysis notebooks and custom code included.
• Data: included in repo or linked if too large.
• Tested, installable library support code.
• Reproducibility support: Makefile and environment.yml
• Good repository practices: README.md, LICENSE, .gitignore.

A “Standard Playbook”

A complete scientific narrative

Backed by analysis details

With supporting tools and tests

Github repo, reproducible on Binder

Made possible by our amazing team!

Thank You!

Scan & fill out if you need to be added to bcourses: https://forms.gle/rqox3mTXp3EGvvZk6