Intro to Reproducibility!

Candace Savonen

Image created by Candace Savonen using Avataars.

R = 0.893

This code runs well on my computer, let me email it to you!

So exciting!

Image created by Candace Savonen using Avataars.

Ruby’s code and data

Image created by Candace Savonen using Avataars.

Re:Re:Re: Data

Hi Ruby, I don’t understand what this code is supposed to be doing...

Re:Re:Re: Data

Hi Avi, It works for me?

Image created by Candace Savonen using Avataars.

R² = 0.893

R² = 0.891

Ruby’s code and data

Image created by Candace Savonen using Avataars.

Data

Code

Results

Ruby the Researcher

Repeatable: keeping everything the same but repeating the analysis - do we get the same results?

Image created by Candace Savonen using Avataars.

Data

Code

Ruby the Researcher

Reproducible: using the same data and analysis but in the hands of another researcher - do we get the same results?

Avi the Associate

Data

Code

Image created by Candace Savonen using Avataars.

New Data

Same Code

Ruby the Researcher

Replicable: with new data do we obtain the same inferences?

Avi the Associate

Variable A and B are positively correlated

Code

Based off of a figure from Essawy et al, 2020 https://doi.org/10.1016/j.envsoft.2020.104753

Effort

Time

Replicability

new researcher, new data

Reproducibility

new researcher, same data

Repeatability

same researcher, same data

Reproducibility saves everyone time and effort!

Image created by Candace Savonen using Avataars.

Ruby’s code

Ruby’s code

Now Ruby

Future Ruby

Image created by Candace Savonen using Avataars.

Ruby’s code - not as reproducible

Image created by Candace Savonen using Avataars.

Ruby’s code - made reproducibly

Reproducibility is a tortoise’s game - it’s an incremental and slow process but it has high payoffs!

Science is a community effort – we each are contributing a domino!

Reproducibility is iterative work!

Image created by Candace Savonen.

Ran once

Re-runs sometimes

Re-runs in every situation and gets the same result every time

Re-runs reliably in most contexts

Perfectly reproducible

​

No analysis reaches here

​

Not repeatable

​

Every analysis starts here

A tale of two differential expression analyses…

Example 1:

Example 2:

Reproducibility != Correctness

​

Reproducibility ~ Consistency

​

But you could be consistently wrong in the same way….

Tips for reproducibility:

1. Write it down! (Documentation!)
2. Organize and re-organize
3. Managing computing environments (Containers)
4. Have everything* publicly available (GitHub)
5. Have others look it over (code review)
6. Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

Now!

Tips for reproducibility:

• Write it down! (Documentation!)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

Tomorrow!

Tips for reproducibility:

• Write it down! (Documentation!)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

Will cover more about this Day 3!

Metadata - the often overlooked but mightily crucial data

What are metadata?

Anything and everything that should be known about your samples!

A B C D

E F G H

I know everything I need to know about these samples from their metadata!

Examples of metadata categories:

​

• Patient/organism of origin
• Patient/organism information
• Demographics
• Disease state
• Treatment state
• Time point (if applicable)
• Processing information
• Batch information
• Processing details (E.g. Isolation methods: Poly-A vs Ribo-minus)
• Anything that should be known about the samples and their handling!

Rules for creating metadata (from Broman & Woo, 2017)

​

Be Consistent

Choose good names for things

Write Dates as YYYY-MM-DD

No Empty Cells

Put Just One Thing in a Cell

Make it a Rectangle

​

1

Rules for creating metadata continued (from Broman & Woo, 2017)

​

Create a Data Dictionary

No Calculations in the Raw Data Files

Do Not Use Font Color or Highlighting as Data

Make Backups

Use Data Validation to Avoid Errors

If you have human data the metadata probably is loaded with PII and/or PHI

Data Organization in Spreadsheets

Tips for reproducibility:

• Write it down! (Documentation)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

A useable, well-documented analysis is more likely to be used and disseminated!

CC-BY by jhudatascience.org

itcrtraining.org/courses

Documentation that every project should have!

1. READMEs
1. Background knowledge
2. Usage info
3. Software requirements to run the thing
4. Basics on how the files are organized

​

Documentation that every project should have!

• READMEs
• Background knowledge
• Usage info
• Software requirements to run the thing
• Basics on how the files are organized
• Code annotations:
• Explain historical decisions
• Explain “quirks” of the code
• Say where more development is needed (TODO)
• Summarize the goals!

​

Use AI tools that are trained for the task you are trying to do!

LLMs can be good at telling you what historical or confusing code is doing

Image created by Candace Savonen using Avataars.

def my_function(x):

result = x

for i in range(10):

for j in range(5):

result = result + 2 * (i + 1) * (j + 1) * (i % 2 == 0 and j % 2 == 0) - 1

return result

Wait, what is this code even?

AI Coding Assistants

Free options!

Getting a basic strategy for how to write code for something

Example prompts:

• How might I go about doing ______ ?
• How could I structure code that would do ______ ?
• Is it possible to create a package that does ______?
• What packages could I use to make code that does _______ ?

Reviewing existing code for improvements

Example prompts:

• Can you tell me how I could make this code more readable?
• Can you help fix the formatting, styling, and indent errors on this code?
• Can you recommend how I could make this code more reproducible?

Annotating or improving documentation

​

​

Example prompts:

• Can you annotate this code?
• Can you explain to me what this code is doing?
• Can you create a README for this code?

https://hutchdatascience.org/AI_for_software/annotating-your-code.html

Tips for reproducibility:

• Write it down! (Documentation)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

• Is project-oriented (Bryan 2017).
• Follows consistent patterns (Shapiro et al. 2021).
• Is easy for you and others to find the files you need quickly (Shapiro et al. 2021).
• Minimizes the likelihood for errors (like writing over files accidentally) (Shapiro et al. 2021).
• Is something maintainable (Shapiro et al. 2021)

Goals for an organizational scheme:

Image created by Candace Savonen.

Chaos reigns - nothing can be found

You lose sleep worrying about your file naming

Perfectly organized but maybe not maintainable

Disorganized and unmanageable

Maintainably organized

Project Organization tips (not one size fits all)

​

Don’t be afraid of folders!

Building Reproducibility Skill Sets!

​

Part 2

Tips for reproducibility:

• Write it down!
• Organize and re-organize
• Managing computing environments
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

final_final_version_100%_up_to_date

final_version_2021

final_version_edit_10_2021

final_version_edit5

final_version_edit4

for_realz_final_edit2

Now was it “final_final_version_100%_up_to_date” or “final_version_edit5” that I was working from?

Course: Intro to Reproducibility and Advanced Reproducibility

final_final_version_100%_up_to_date

final_version_2021

final_version_edit_10_2021

final_version_edit5

final_version_edit4

for_realz_final_edit2

Now was it “final_final_version_100%_up_to_date” or “final_version_edit5” that I was working from?

Course: Intro to Reproducibility and Advanced Reproducibility

VERSION CONTROL

NEEDED!

Image created by Candace Savonen using Avataars

“Code and data are available upon request by email.”

I’m requesting the code from your recent paper in the journal....

The corresponding author’s inbox

Course: Intro to Reproducibility and Advanced Reproducibility

Image created by Candace Savonen using Avataars

“Code and data are available upon request by email.”

I’m requesting the code from your recent paper in the journal....

The corresponding author’s inbox

Course: Intro to Reproducibility and Advanced Reproducibility

OPEN SOURCE

NEEDED!

Open source! Code is publicly available for reuse and repurpose

Course: Intro to Reproducibility and Advanced Reproducibility

Oh no! My computer broke! – Good thing Github has all my work.

Back Ups! For the worst case scenario

Course: Intro to Reproducibility and Advanced Reproducibility

Why did we write the code this way? I don’t remember… Good thing through git tracking I can look into this file’s history and remind myself how it became this.

Keep better records! Cut down on refinding the same thing

Course: Intro to Reproducibility and Advanced Reproducibility

time/work

Keep the “perfected” version safe! Free to experiment with easy restore capabilities

Course: Intro to Reproducibility and Advanced Reproducibility

main

Avi’s changes added

Avi’s and Ruby’s changes added

Easier to work simultaneously with teammates!

​

No emailing files back and forth

Course: Intro to Reproducibility and Advanced Reproducibility

Code is best done as a team activity!

​

​

​

Code is best done as a team activity!

​

If this isn’t an option – at least a fake person: aka AI

Why GitHub?

• Open Source - Transparency and helping the community by sharing!

​

• Version Controlled - Better records; better back ups; better teamwork!

​

• Free

​

• Popular (Bitbucket is fine too)

What’s a pull request?

​

A way to propose changes so that they can be discussed before they are incorporated

https://github.com/FredHutch/gimap/pull/33

Pull request model

Direct changes

Less verification

​

Knowledge silos more likely to happen

​

​

Mistakes may infiltrate the public version

​

Records of the decisions made may be more diffuse or nonexistent

More eyes to check the changes

​

More people know what’s happening

(Knowledge transfer)

​

​

Better chance of catching mistakes sooner

​

Better records of communication about the analysis/software

Intro to GitHub -

How to file a pull request!

Tips for reproducibility:

• Write it down! (Documentation!)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

We don’t work alone!

​

Sometimes the collaborator with questions is “Future You”.

​

Readability >>>>> Cleverness

Course: Intro to Reproducibility and Advanced Reproducibility

Image created by Candace Savonen using Avataars.

Ruby’s code

Ruby’s code

Now Ruby

Future Ruby

AI tools make it so even if you are a “lone programmer” you can still have a code review!

R = 0.893

This code runs well on my computer, let me email it to you!

So exciting!

Course: Intro to Reproducibility and Advanced Reproducibility

Main goals of an original code author

​

1. Set up your reviewer for success by erring toward overcommunicating

​

• Interpret reviews positively!

​

• Determine solutions collaboratively.

Course: Intro to Reproducibility and Advanced Reproducibility

Ruby wants to merge 5 commits into main from a-new-branch

Reviewers

Avi

Code changes #10

• 999100057849 - 1901

1870

0

10

2

Ruby requested a review from Avi 10 minutes ago

Course: Intro to Reproducibility and Advanced Reproducibility

Ruby wants to merge 5 commits into main from a-new-branch

Reviewers

Avi

Update documentation for heatmap-script.R

• 50 - 19

1

0

2

2

Ruby requested a review from Avi 10 minutes ago

Course: Intro to Reproducibility and Advanced Reproducibility

Image from https://phauer.com/2018/code-review-guidelines/

Main goals of a reviewer

​

• Identify opportunities for learning and improvement.

​

• Communicate these with positivity and empathy.

​

• Determine solutions collaboratively.

Course: Intro to Reproducibility and Advanced Reproducibility

Empathy is an important part of effective code review!

Image from https://quickbirdstudios.com/blog/code-review-best-practices-guidelines/

Remember the author of the pull request has been putting time and effort into this!

Course: Intro to Reproducibility and Advanced Reproducibility

Ruby requested a review from Avi 10 minutes ago

Course: Intro to Reproducibility and Advanced Reproducibility

Ruby requested a review from Avi 10 minutes ago

Course: Intro to Reproducibility and Advanced Reproducibility

Tips for reproducibility:

• Write it down! (Documentation!)
• Organize and re-organize
• Managing computing environments (Containers)
• Have everything* publicly available (GitHub)
• Have others look it over (code review)
• Automate monotonous stuff (CI/CD)

​

​

*Everything that doesn’t violate IRB privacy and ethical data handling guidelines

CI/CD:

​

Continuous Integration/

Continuous Deployment

​

– A software concept that is useful for reproducible data analyses!

Course: Github Automation for Scientists

Every time a change is proposed we check it before incorporating it

​

​

​

Change made Run a series of tests

CI/CD: Continuous Integration/ Continuous Deployment

Course: Github Automation for Scientists

I will re-run this analysis whenever changes are made to it.

Course: Github Automation for Scientists

Think smarter not harder: automation in a nutshell

Course: Github Automation for Scientists

Continuous integration / continuous deployment

You’re a construction manager

​

Should you:

​

1. Check that your construction plans are good and meet safety and engineering standards as you build it?

OR

• Build the entire building without consulting anyone and only have them check these things after you are done and its basically a demo job if you want to fix it?

Course: Github Automation for Scientists

Problems accumulate without using CI/CD

Time/Effort

Re-run

Re-run

A bug being introduced

3 bugs to track down!

Course: Github Automation for Scientists

Catching changes/problems early with CI/CD

Time/Effort

Re-run

Re-run

Re-run

Re-run

1 bug

1 bug

1 bug

Course: Github Automation for Scientists

Catching changes/problems early with CI/CD

R = 0.902

R = 0.902

R = 0.905

R = 0.902

Time/Effort

Re-run

Re-run

Re-run

Re-run

Course: Github Automation for Scientists

This is only the beginning!

• Intro to Reproducibility
• DRY code
• Data organization
• GitHub intro
• Advanced Reproducibility
• Intro to Containers
• Intro to GitHub Actions
• GitHub intermediate
• Code review
• Github Automation for Scientists
• Principles of data sharing
• CI/CD principles in science and analyses
• GitHub Actions + Docker
• Containers for Scientists STILL UNDER DEVELOPMENT
• https://hutchdatascience.org/containers-for-scientists-sandbox/
• Pulling and modifying images
• Using containers
• Sharing images
• Troubleshooting

This is only the beginning!

• Intro to Reproducibility
• DRY code
• Data organization
• GitHub intro
• Advanced Reproducibility
• Intro to Containers
• Intro to GitHub Actions
• GitHub intermediate
• Code review
• Github Automation for Scientists
• Principles of data sharing
• CI/CD principles in science and analyses
• GitHub Actions + Docker
• Containers for Scientists STILL UNDER DEVELOPMENT
• https://hutchdatascience.org/containers-for-scientists-sandbox/
• Pulling and modifying images
• Using containers
• Sharing images
• Troubleshooting