1 of 53

Daniel Nüst�Institut für Geoinformatik�d.n@wwu.de | @nordholmen�0000-0002-0024-5046

Slides: http://bit.ly/hangout21-repro

1

Practical reproducibility and reproducibility vs. peer review

�Spatial Data Science Hangout, spatial@ucsb, Spring ‘21

2 of 53

2

https://giphy.com/gifs/usnationalarchives-nasa-scientist-scientists-1F1JGyGZhiSAA8Vuhn

https://theconversation.com/how-computers-broke-science-and-what-we-can-do-to-fix-it-49938

https://giphy.com/gifs/with-computers-fascination-PxSFAnuubLkSA

https://giphy.com/gifs/david-hasselhoff-M3o3fL9nnxG4o

CC-BY 3.0, Sebastian Bertalan, Wikimedia Commons

An article about computational science in � a scientific publication is not the � scholarship itself, it is merely advertising

of the scholarship. The actual scholarship

is the complete software development � environment and the complete set of � instructions which generated the figures.

https://doi.org/10.1190/1.1822162

https://doi.org/10.1007/978-1-4612-2544-7_5

Claerbout’s claim:

WAS IST DAS PROBLEM? Wissenschaftler benutzen Computer.�Und zwar immer mehr, weil Computer uns helfen können ansonsten unlösbare Fragen zu beantworten, wie bereits im Grußwort heute morgen festgestellt wurde.

ABER die Zusammenarbeit mit unseren digitalen Werkzeugen ist oft nicht reibungslos! Das ist kein Fehler der Wissenschaftler allein, denn unsere virtuellen Labore sind komplizierter als man denkt.

Der Archäologe Ben Marwick stellte folgendes fest:�moderne Wissenschaft ist so kompliziert und Publikation in Zeitschriften so eingeschränkt, dass Artikel gar nicht alle Details beinhalten können um die genutzten Methoden und die getroffenen Entscheidungen nachhaltig zu dokumentieren. Diese Bruchstelle in der Wissenschaftskommunikation wird immer relevanter.

Zu lange haben wir nur gesagt: guckt mal, so sieht das bei mir auf dem Bildschirm am Ende aus. Jedoch kann in den wenigsten Fällen wirklich hingeschaut oder gar hineingeschaut werden! Gutachter evaluieren selten Daten und Code weil Zeit, passende Expertise und Richtlinien fehlen.

[Unter den ersten, die diese Bruchstelle auf den Punkt brachten, waren Buckheit und Donoho im Jahr 1995 (in meiner Übersetzung):

“Ein Artikel über rechentechnische Wissenschaft ist nicht das Wissen selbst, es ist nur die Werbung. Die tatsächliche Wissenschaft umfasst die komplette Softwareentwicklungsumgebung und die gesamten Handlungsanweisungen welche die Darstellungen erzeugten”.

Das Kernproblem ist also, dass es keine Tradition oder etablierte Praxis gibt, wie alle Bausteine, die Wissenschaftler_innen für ihre Forschung verwenden, so veröffentlicht werden, dass sie komplett nachvollziehbar und wiederverwendbar sind. Wie soll dann die tatsächliche Wissenschaft begutachtet werden?]

3 of 53

Crisis? Crisis of what?

Credibility crisis?�Replicability crisis?�Reproducibility crisis?�Robustness crisis?�Generalisability crisis?

3

https://doi.org/10.1073/pnas.1708272114

https://the-turing-way.netlify.app/reproducible-research/overview/overview-definitions.html

Is there a crisis? And what kind of crisis? Depending on who you ask, there is a credibility crisis, replicability crisis, or reproducibility crisis, or all of them.

I agree with Daniele Fanelli that the crisis perspective might not be the best approach. In her worth reading opinion piece she argues that there undoubtedly is a huge transformation going on which we can use to introduce positive change. I agree it is much more important that there is broad acknowledgement that scholarly communication and the way we share our work must change and that there is a call to action by leaders in the fields.

To be clear about the distinction between replicability and reproducibility: I am not talking about replicability at all, today, that’s a bigger problem to solve, because it involves more resources and even irreproducible papers in my opinion have the potential to be replicated - because testing the original hypothesis is always possible, because at least the hypothesis and the findings can be effectively communicated via paper. I see reproducibility as a step towards selectively and consciously working towards replications. Also, in general, replications and reproductions should be accepted as academic outputs.�But reproducibility can also be all we can get, if observations or infrastructure is unique. In the Earth sciences, for example, we’re not going to shoot another hundred satellites up into the orbit just to see that the existing ones work. In social sciences, we will not start a new social network and grow it to the size of Twitter so replicate research on human online interactions.

Important for today: reproducibility is something that can actually be determined as part of a peer review process!

-----------

https://giphy.com/gifs/season-3-the-simpsons-3x5-3o6MbqW1L46tHJUlYQ

The different dimensions of reproducible research described in the matrix above have the following definitions:

Reproducible: A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.
Replicable: A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.
Robust: A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question (for example one pipeline written in R and another written in Python) and a qualitatively similar or identical answer is produced. Robust results show that the work is not dependent on the specificities of the programming language chosen to perform the analysis.
Generalisable: Combining replicable and robust findings allow us to form generalisable results. Note that running an analysis on a different software implementation and with a different dataset does not provide generalised results. There will be many more steps to know how well the work applies to all the different aspects of the research question. Generalisation is an important step towards understanding that the result is not dependent on a particular dataset nor a particular version of the analysis pipeline.

More information on these definitions can be found in “Reproducibility vs. Replicability: A Brief History of a Confused Terminology” by Hans E. Plesser [Ple18].

4 of 53

Practical Reproducibility in Geography and Geosciences

Daniel Nüst & Edzer Pebesma (2020) Practical Reproducibility in Geography and Geosciences, Annals of the American Association of Geographers, DOI:10.1080/24694452.2020.1806028�PDF: http://nuest.staff.ifgi.de/N%C3%BCst-and-Pebesma_2020_AAM_Practical-Reproducibility-in-Geography-and-Geosciences.pdf

Creating reproducible workflows

Computing environment: hardware + software, containers/virtualisation (Binder), freezing/pinning�Script-based workflows: no point-and-click GIS, notebooks (Jupyter, R Markdown)�> Research compendium

Challenges

Education, publishing practices, SDIs, GIS, proprietary software,�lack of rewards/pressure, sensitive data, time, ...�> all solvable

5 of 53

Reproducible Research &�research software

Peer Review

5

Reproducible research and peer review are cornerstones of science. But are they getting along?

6 of 53

CODECHECK

7 of 53

7

The inverse problem in reproducible research. Figure 1 of https://doi.org/10.12688/f1000research.51738.1

The left half of the diagram shows a diverse range of materials used within a laboratory. These materials are often then condensed for sharing with the outside world via the research paper, a static PDF document. Working backwards from the PDF to the underlying materials is impossible. This prohibits reuse and is not only non-transparent for a specific paper but is also ineffective for science as a whole. By sharing the materials on the left, others outside the lab can enhance this work.

8 of 53

8

The CODECHECK example process implementation. Figure 2 of https://doi.org/10.12688/f1000research.51738.1

The left half of the diagram shows a diverse range of materials used within a laboratory. These materials are often then condensed for sharing with the outside world via the research paper, a static PDF document. Working backwards from the PDF to the underlying materials is impossible. This prohibits reuse and is not only non-transparent for a specific paper but is also ineffective for science as a whole. By sharing the materials on the left, others outside the lab can enhance this work.

https://codecheck.org.uk/process/

9 of 53

9

Independent execution of computations underlying research articles.

One re-execution by codechecker during peer review

👶

Codecheckers record but don’t investigate or fix.
Communication between humans is key.
Credit is given to codecheckers.
Workflows must be auditable.
Open by default and transitional by disposition.

25 Certificates https://codecheck.org.uk/register/

📃

🔏

Eine Idee um dem Mangel an Reproduzierbarkeit zu begegnen ist

CODECHECK.

Ein CODECHECK bedeutet, dass eine Person, der “Codechecker”, an einem Zeitpunkt in der Lage ist, das Material, was der Autor zur Verfügung stellt, nachzuvollziehen und den Arbeitsablauf ohne nennenswerte Fehler einmalig auszuführen. Dies ist ein bewusst sehr eingeschränkter Umfang mit geringer Einstiegshürde für Autoren.�Die Idee ist ohne Standards oder Infrastruktur umsetzbar.

Ein CODECHECK kann zu unterschiedlichen Zeitpunkten durch verschiedene Rollen durchgeführt werden, jedoch könnten insbesondere

Akademiker_innen am Beginn ihrer Karriere ihre Expertise rund um digitale Methoden in die Wissenschaftskommunikation und das Peer Review einbringen.

Als Ergebnis wird ein CODECHECK-Zertifikat zitierfähig veröffentlicht. Wie andere Gutacher steht der Codechecker für sein Urteil ein.

Die Grundlage von CODECHECK bildern vier Prinzipien welche definieren, was ein Codechecker leisten kann und wie wir mit mehr Vertrauen sagen können:

The workflow and variations outlined describe our current views on how code could be checked. They are not immutable, but we believe the following core principles underpin our CODECHECK process:

1. Codecheckers record but don’t investigate or fix.

The codechecker follows the author’s instructions to run the code. If instructions are unclear, or if code does not run, the codechecker tells the author. We believe that the job of the codechecker is not to fix these problems but simply to report them to the author and await a fix. The level of documentation required for third parties to reproduce a workflow is hard to get right, and too often this uncertainty leads researchers to give up and not document it at all. The conversation with a codechecker fixes this problem.

2. Communication between humans is key.

Some code may work without any interaction, e.g. 21, but often there are hidden dependencies that need adjusting for a particular system. Allowing the codechecker to communicate directly and openly with the author make this process as constructive as possible; routing this conversation (possibly anonymously) through a publisher would introduce delays and inhibit community building.

3. Credit is given to codecheckers.

The value of performing a CODECHECK is comparable to that of a peer review, and it may require a similar amount of time. Therefore, the codechecker’s activity should be recorded, ideally in the published paper. The public record can be realised by publishing the certificate in a citable form (i.e., with a DOI), by listing codecheckers on the journal’s website or, ideally, by publishing the checks alongside peer review activities in public databases.

4. Workflows must be auditable.

The codechecker should have sufficient material to validate the workflow outputs submitted by the authors. Stark²² calls this “preproducibility” and the ICERM report¹¹ defines the level “Auditable Research” similarly. Communities can establish their own good practices or adapt generic concepts and practical tools, such as publishing all building blocks of science in a research compendium (cf. https://research-compendium.science/) or “repro-pack”²³. A completed check means that code could be executed at least once using the provided instructions, and, therefore, all code and data was given and could be investigated more deeply or extended in the future. Ideally, this is a “one click” step, but achieving this requires particular skills and a sufficient level of documentation for third parties. Furthermore, automation may lead to people gaming the system or reliance on technology, which can often hide important details. All such aspects can reduce the understandability of the material, so we estimate our approach to codechecking, done without automation and with open human communication, to be a simple way to ensure long-term transparency and usefulness. We acknowledge that others have argued in favour of bitwise reproducibility because, in the long run, it can be automated (e.g., https://twitter.com/khinsen/status/1242842759733665799), but until then we need CODECHECK’s approach.

5. Open by default and transitional by disposition.

Unless there are strong reasons to the contrary (e.g., sensitive data on human subjects), all code and data, both from author and codechecker, will be made freely available when the certificate is published. Openness is not required for the paper itself, to accommodate journals in their transition to Open Access models. The code and data publication should follow community good practices. Ultimately we may find that CODECHECK activities are subsumed within peer review.

10 of 53

10

Nüst D and Eglen SJ. CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility [version 1; peer review: awaiting peer review]. F1000Research 2021, 10:253 (https://doi.org/10.12688/f1000research.51738.1)

11 of 53

Reproducible AGILE

12 of 53

AGILE Reproducible Paper Guidelines 🇬🇧 🇪🇸

https://doi.org/10.17605/OSF.IO/CB7Z8

Created by AGILE Initiative in 2019, see report at https://osf.io/hupxr/

Transparency & Reproducibility�GIScience�https://osf.io/phmce/wiki/home/

Promotion�Acknowledge spectrum

12

At the core of the AGILE reproducibility review are the AGILE Reproducible Paper Guidelines. They are are the first compilation of tips and tools how to increase transparency and reproducibility in GIScience papers.

They are complemented with a Wiki with examples for typical GIScience datasets and workflows.

The guidelines’ intention is to promote positive examples of reproducible computational research and to acknowledge the full spectrum of reproducibility as well as the transition towards establishment of reproducible research practices.

The only requirement of the guidelines is that each paper must include a Data and Software Availability sub-section, or “DASA section” for short. It is absolutely possible not to provide access to data and code and to still comply with the guidelines at the same time, as long as there are good reasons given.

13 of 53

The guidelines

Author guidelines�Data in Research Papers�Computational workflows �in Research Papers�Pre-submission checklist�Writing DASA section

Rationale/Motivation/Vision

Reviewer guidelines (what not to worry about)

Reproducibility reviewer guidelines

13

https://doi.org/10.17605/OSF.IO/CB7Z8

14 of 53

AGILE conference review process

Proceedings:�https://www.agile-giscience-series.net/review_process.html

Process documentation:�https://osf.io/7rjpe/

14

Reproducibility review after accept/reject decisions, triggered by regular reviewer

Reproducibility review & communication

Community conference

Badges on proceedings page

Presentation at conference

Read full report at https://osf.io/7rjpe/

First a few words about the process. You can find the details in the linked document.

Let’s start with the order: the reproducibility review happened after the authors were notified about the acceptance or rejection. Ideally, we wanted to review only papers that were identified by the scientific reviewers, but we found that that was not reliable so we briefly checked all papers ourselves. Also, none of the regular reviewers could be intrigued to attempt a reproduction.

This reduced the number of papers and allowed us to communicate directly with the authors during the reproducibility reviews.

The process was not smooth, because AGILE is a community-led conference managed by volunteers, and the authors are the ones who prepare the camera-ready copies of the articles. So there is a lot of moving parts with no central control. On top of that, the conference switched the publisher and the Coronavirus destroyed all schedules.

Eventually, the papers were published. The publisher added badges on the landing pages of the proceedings, which link to the reproducibility reports, as you can see on the right hand side. The papers themselves do not reference the reports.

We originally planned to have short presentations of selected reproducible papers in the final conference session, where candidates for the the best paper award are presented and voted live by the audience. Unfortunately, the in-person event was cancelled, so we could not have a showcase of the first batch of reproducible papers as we would have liked to.�

15 of 53

Reproducibility review results

6 reproducibility reports published in 2020�🔥 9 more coming for 2021

16 (2020) not possible/not attempted (5 of which after communication with authors):

no starting point in the paper
documentation insufficient for third party
sensitive/confidential/commercial data
proprietary software
software paper
conceptual papers

15

https://osf.io/6k5fh/

Six reproductions were partially successful; “Partially”, because not all details of the computational results, for example not all figures, were reproducible.

16 papers were assessed as impossible to reproduce early on, for different reasons. Here are barriers we faced:

If authors provided no starting point, that is no link to data or code at all, we did not attempt a reproduction even if there clearly was a computational workflow; we neither contacted the authors in these cases;

Often documentation was not detailed enough for a third party to reproduce a workflow, for example generic dataset citations or manual steps, especially for data download, or anonymisation that was not removed in a timely manner;

Some papers use sensitive data on human subjects, or confidential or commercial data, which we did not have access to;

Some papers relied on proprietary software;

One paper was simply describing a piece of software, which is fine, but didn’t give us anything to evaluate;

similarly, in previous years of the conference, we observed purely conceptual papers, which of course are also not computationally reproducible.

In all of these cases, authors could have documented the reason themselves, for example in the DASA section, but few did.

16 of 53

What can you do today?

16

17 of 53

Reproducible Research & Open Science

https://doi.org/10.1126/science.1213847

https://www.nature.com/articles/d41586-018-05256-0

18 of 53

18

Quintana, D. S. (2020, November 28). Five things about open and reproducible science that every early career researcher should know. https://doi.org/10.17605/OSF.IO/DZTVQ

19 of 53

What can scientists do?

Take one step at a time.

Create and publish Research Compendia�(Your code is good enough!):�https://research-compendium.science/

Become a codechecker or reprohacker.

Strive to be an open science champion especially�if you’re junior in your field. We need to be the change, find communities.�[RIOT Science Club Talk by Gavin Buckinham; preprint by Sam Westwood]

19

20 of 53

Thanks!

Daniel Nüst�Institut für Geoinformatik�d.n@wwu.de | @nordholmen�0000-0002-0024-5046

Slides: http://bit.ly/hangout21-repro

All images https://giphy.com/

21 of 53

Bonus slides for discussion

22 of 53

Reproducible AGILE and CODECHECK:�Highlights of Lessons learned

Spectrum or layers of reproducibility

Effect of guidelines at AGILE: improved reproducibility

Reproducibility reports/CODECHECK certificates full of recommendations for improvement, well received by authors, even included in revision before publication

Good practices spread slowly, establishing a process is tedious

Challenges for reproducibility reviewer: Inconsistencies and disconnects (figures), lack of documentation, unknown runtimes vs. no subsets of data, lack of repro guidance

Reproductions are rewarding and educational, matching expertises tricky

Safety net (👀), not security

22

Read full report at https://osf.io/7rjpe/

Highlights of things that we learned during the first reproducibility review at an AGILE conference and the 23 codechecks we did so far.

We saw the full spectrum of reproducibility, which encourages us to continue with our current approach of positive encouragement and education.

At AGIlE, we saw that compared to previous years’ submissions, the guidelines did effectively increase community awareness and improved reproducibility. Of course we also looked harder than before, but we are still very happy with the overall outcome of the reproductions.

The reproducibility reports have a lot of details with recommendations how to improve workflows. Some authors picked these up pretty quickly and even improved their material in time with the publication of the articles, which I think is a great reward for the reproducibility reviewers. I hope the reports were as friendly as the ReproHackers’ feedback!

Also, good practices spread slowly because people need to change their (even daily) habits; at AGILE this led to reviews being less strict than originally anticipated. I had to leard that slow change can be a good thing because you can keep everybody on board. Organising the whole process is hard, and needs improvements over time - so I try to worry less about that.

The challenges for reviewers respectively codecheckers are manifold, but mostly easy to solve in a bilateral conversation.

While everyone should be able to reproduce any workflow, matching reviewer expertise can be tricky yet important for effective reviews - in any case I personally found the reproductions to be always educational and a rewarding task.

And finally, there are of course limitations. The AGILE reproducibility review and CODECHECK provide safety nets, a layer to make sure we are not making any avoidable mistakes such as incomplete datasets. They do not provide security against fraud or malicious activities, though they can ensure that future investigators have all they need to dig deeper.

----------

https://www.quora.com/What-is-the-difference-between-safety-and-security�Safety is the prevention of accidents (accidents which may or may not involve human agents, but are in any case not intentional).�Security is the prevention of malicious activities by people (mugging, burglary, robbery, terrorist activities, etc.).

Avoid inherent dangers (driving sober with a regularly inspected car), not protect against external factors (tree falling onto the road, someone tampering with the breaks).

23 of 53

How does the future of reproducible research in peer review look like?

Reproducibility is possible, but disciplines/communities must agree what “peer review” entails and acknowledge the efforts (ECRs, RSEs) in a positive way.

Help each other! Move together as a community through disruptive changes.�Then reproducible research and peer review will get along just fine.

23

How does the future of reproducible research in peer review look like?

So there are many things we can do as individuals and groups, and I tried to introduce you to two examples how we can connect reproducible research and peer review.�Each community or disciplines needs to have a conversation about what peer review entails. This should be a moving target, increasing requirements over time. I presented two approaches today that focus on execution of workflows and communication between authors and reviewers.�In my opinion, there is no alternative to enforcing reproducibility to ensure scientific results remain trustworthy and scientists collaborate more effectively. Even if that means that we have to fix some of science’s biggest problems around incentives and evaluation in the process. We also have to get more experts involved even if they are traditionally not peer reviewers.�Focussing on positive rewards can help to smooth the process, because

we shine a light on the champions of open and reproducible research.

By making each other shine, we all look better.

Cultural change is a team effort and it does not hurt even it is a little disruptive sometimes.

Then reproducible research and peer review can get along, they will even help each other! Maybe preprint codechecking of workflows to be collected in overlay journals will be natural for researchers in ten years?

https://giphy.com/gifs/wLDXxrBcH4FPO �https://giphy.com/gifs/goheels-carolina-unc-go-heels-3osxYaWWuMvKmJ6j3q/media �https://giphy.com/gifs/hug-love-winnie-the-pooh-llmZp6fCVb4ju

24 of 53

https://codecheck.org.uk/register/

24

Independent execution of computations underlying research articles.

25 of 53

25

26 of 53

26

J. Leek’s tidypvals

https://simplystatistics.org /2017/07/26/announcing-the-tidypvals-package/

“Notice�Anything�funny?”

27 of 53

The many problems of science

Publish or perish�Broken metrics (citations, JIF)�Structural change not considering � senior academics�Publication bias�Long-term funding for tools & infrastructure�HARKing�p-Hacking�Scholarly communication 1.0�Lack of reusability�Lack of transparency�Lack of reproducibility�Reinventing the wheel�Retraction practices�Not invented here syndrome�Fraud�Imposter syndrome�No “negative” citation�...

Open Science (OER, OA, OS, OPR)�Registered reports/preregistration�Altmetrics�Preprints�Leiden Manifesto�DORA�Vienna Principles�Citing data and software�Software papers�Data and software as products of research�RSEng & RSEs (software sustainability)�CRediT�Research Compendia�Ten Hot Topics Around Scholarly Publishing�Code review (PyOpenSci, ROpenSci, JOSS)�...�

27

https://doi.org/10.31234/osf.io/p6e9c

https://simplystatistics.org/2017/07/26/announcing-the-tidypvals-package/

https://giphy.com/gifs/bbcamerica-cute-animals-lifestory-Ze3RpHue7qkwvcYOOf

CULTURAL�CHANGE

28 of 53

28

https://doi.org/10.1186/s13059-015-0850-7

reproducibility helps to avoid disaster
reproducibility makes it easier to write papers
reproducibility helps reviewers see it your way
reproducibility enables continuity of your work
reproducibility helps to build your reputation

29 of 53

30 of 53

Traditional and modern scientists

30

T

https://en.wikipedia.org/wiki/T-shaped_skills

https://doi.org/10.1007/s10816-015-9272-9

https://jakevdp.github.io/blog/2014/08/22/hacking-academia/

https://www.sciencemag.org/careers/2013/05/when-all-science-becomes-data-science

https://escience.washington.edu/community-level-data-science-and-its-spheres-of-influence-beyond-novelty-squared/

Π

Broad knowledge: across disciplines�collaborate with other experts, apply outside of own field

Deep knowledge: expertise and skills within a single field

Computer & method skills�statistics, reproducibility, programming, data science

[no speaker notes]

----------------------------

https://en.wikipedia.org/wiki/T-shaped_skills

The concept of T-shaped skills, or T-shaped persons is a metaphor used in job recruitment to describe the abilities of persons in the workforce. The vertical bar on the letter T represents the depth of related skills and expertise in a single field, whereas the horizontal bar is the ability to collaborate across disciplines with experts in other areas and to apply knowledge in areas of expertise other than one's own.

https://escience.washington.edu/community-level-data-science-and-its-spheres-of-influence-beyond-novelty-squared/

The idealized characterization of data science in academia is also represented in the idea of shifting from the traditional T-shaped scientists, who have deep expertise in a single domain, to Π (Pi)-shaped scientists with deep expertise in both a domain and methodological science (as coined by Alex Szalay and discussed here and here). [...]

We encountered lots of data science, emerging across a network of interactions across a multitude of differently shaped scientists, from T to Γ (Gamma) to Π to even Μ (Mu) -shaped! Individuals mostly did not see themselves as particularly Π-shaped, a category many reserved for only a very select few that had accomplished an exceptional level of expertise and contribution to multiple fields.

By participating in the RSE community, you are on your path to become a modern pi-shaped scientists!

However, Pi-shaped people only get so far on their own! At some level of complexity, professionalisation is important.

31 of 53

Code Review

31

Code Review Community Working Group

Boettiger, C., Chamberlain, S., Hart, E., & Ram, K. (2015). Building Software, Building Community: Lessons from the rOpenSci Project. Journal of Open Research Software, 3(1), e8. doi:10.5334/jors.bu

32 of 53

Reproducible computational research in journals & conferences

32

33 of 53

Findings

Overall

Saw full spectrum of reproducibility
Compared to previous years’ submissions, the guidelines and increased community awareness markedly improved reproducibility
⅚ reproduced papers have DASA; all embrace guidelines
Reproducibility reports with many recommendations for improvement, well received by authors, even included in revision before publication > reward!
Good practices spread slowly
Process

33

Read full report at https://osf.io/7rjpe/

This is the first reproducibility review at an AGILE conference, and as you can expect from a first time, we deviated quite a lot from the original plan. Here is what we learned:

We saw the full spectrum of reproducibility, which encourages us to continue with our current approach.

We saw that compared to previous years’ submissions, the guidelines did effectively increase community awareness and improved reproducibility. Of course we also looked harder than before, but we are still very happy with the overall outcome of the reproductions.

Almost all reproduced papers have the DASA section, and all follow them at least in part. That’s a great sign that even recommended guidelines have impact.

The reproducibility reports have a lot of details with recommendations how to improve workflows. Some authors picked these up pretty quickly and even improved their material in time with the publication of the articles, which I think is a great reward for the reproducibility reviewers.

Also, good practices spread slowly because people need to change their (even daily) habits; this also led to us being less strict than originally anticipated, but that’s a good flexibility to let the community of authors influence the pace of change, too. Slow change can be a good thing.

And finally, the process cannot be established at once, but needs to be improved over time, especially the coordination with other conference committees and the publisher. For example, we plan to not allow authors to object the publication of the reproducibility report anymore. Since only successful reproductions will be published anyway, we don’t see a need for this additional step. The need to credit the reviewer also outweights possible author’s concerns here.

----------

Image: https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Linear_visible_spectrum.svg/1024px-Linear_visible_spectrum.svg.png

34 of 53

Findings

Challenges for reproducibility reviewer:

Inconsistencies (identifiers, links) between paper and code
Lack of connections between artefacts (code <> figure)
Workspaces layout: no documentation, absolute paths
Unknown runtime and no demo subsets of data
No guidance on efforts and stop points

�All efforts beyond mere workflow execution

34

Read full report at https://osf.io/7rjpe/

These are the most common challenges that the reproducibility reviewers faced:

We found inconsistencies between the versions and identifiers used in the data deposit and the manuscript. That is one of the main reasons why proper depositing (i.e., with a DOI) of data and code are so important.

On top of that, a lack of connections between figures in the paper and documentation/code really slowed reviewers down. If you are an author, please use the figure numbers in your README and file names. Ideally, mention the command or script file in the figure description, too.

Workspace layout and methods often were not accessible for third parties: no clear README file, usage of absolute paths; these are things that the guidelines do cover, but authors seem to be unaware how tedious these are to figure out.

Another challenge is undocumented run time, so it can happen that you start a process and after 24 hours it tells you there is 10% progress; authors need to be aware of this and provide subsets for quicker evaluation of methodologies.

This is related to the challenge that due to the lack of experiences, we did not provide any guidance on how much time reviewers should spend before stopping. I estimate that we spent 4 to 10 person hours per review this year, not including running the actual computations, which is not feasible in the long term.

To summarise, I would say that these are all problems outside of the mere execution of the workflow. This is where authors can do a better job to reduce the time needed to understand their method by preparing more reproducible workflows. Trying to attempt a reproduction yourself could be a great way to increase mutual understanding.

----------

Image: https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Linear_visible_spectrum.svg/1024px-Linear_visible_spectrum.svg.png

35 of 53

🙌

35

How to put your community on a path towards�more reproducibility in 5 easy hard steps

Build a team of enthusiasts (workshop, social events)
Assess the current state and raise awareness (workshop, paper)
Institutional support (🙏 AGILE Council 🙏 + committee chairs)
Positive encouragement (no reproduction != bad science)
Keep at it!

In just three years, we were able to plant the seed of cultural and structural change (if I myself may say so). So here is how to put your community on a path towards more reproducibility in 5 easy steps:

Build a team of enthusiasts, for example by organising a workshop
Assess the current state and talk about it - this is how people will realise you are serious; in our case it was pre-conference workshops and a paper with a strong message
Try to get institutional support - in our case the support of the AGILE Council and the conference committee chairs made everything possible
Try to get the community on your side by staying positive and inclusive - irreproducibility is usually not a sign for bad science, but for a lack of knowledge and incentives, which you can try to create
Keep at it, cultural change takes time - putting reproducibility on the map is an achievement in itself, and every little step counts; providing feedback as part of the reproducibility reviews will, one author at a time, educate your community.

36 of 53

What can communities and institutions do?

Introduce reproducibility reviews - CODECHECK (or not)!

Workshops on RCR, ReproHacks

Provide support (R2S2, Anja)

Rewards and incentives

�Awareness > Change

36

https://giphy.com/gifs/chicagodancecrash-KCqjrcPfL55q3MkgHZ

Introduce reproducibility reviews, top down if you’re an editor, or buttom up if your’re author or reviewer. Just do it, it will shift practice over time.

Become and advocate and educator! You can organise a workshop or a ReproHack, and help more and more community members build up the expertise to work more reproducibly. A lack of reproducibility is rarely intentional or supported by a strong opinion, but a lack of knowledge and rewards.

Create incentives and rewards.

Awareness leads to change. In the AGILE community, the team behind Reproducible AGILE was able to get from zero to six reproducible papers in three years, which I’m really proud of. However, I must admit this was only possible because we had institutional support.�We’re still working hard to find journals who want to establish CODECHECKs.

One tiny step at a time is still progress, and as this dancer shows us, you can still be graceful and stylish taking small steps.

37 of 53

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/ZTC7M

https://doi.org/10.17605/OSF.IO/7XRQG

37

38 of 53

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/XS5YR

https://doi.org/10.17605/OSF.IO/SUWPJ

38

39 of 53

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/7TWR2

https://doi.org/10.17605/OSF.IO/5SVMT

39

[JUST SHOW SLIDE]

On the left, we have a paper that used an anonymous supplement, but needed some back and forth to get the workflow right; the authors provided a textual definition of the computing environment, which the reproducibility reviewer turned into a standardised format; eventually the plots were pretty close to originals from the paper, though you can see here that the axes labels are illegible in the reproductions.

On the right hand side, we have a paper with data and code on OSF; with a lot of trial and error with some author help, eventually we got a subset of the workflow running, because the full analysis was much too large; the recreated figures were close enough to the original to trust everything is there, though there was no perfect match (for example, different figure titles).

Please take a look at these reports and let me know if you think they are useful for authors, students, readers, or reviewers, and how we can improve them. So what do we do with these reports?

40 of 53

The guidelines for

reproducibility reviewers (WIP)

Ideal vs. realistic

Role

Skills

Do’s & dont’s

40

https://docs.google.com/document/d/1Kc-ToUVcrdsq6aB8Qy2J_rIluFwDniv6GHGtZuPvIEo/edit#

[4:00]��I will skip the details of the author guidelines, but I’d like to take a look at the brand new reproducibility reviewer guidelines, because they really set the tone and scope of how we do reproducibility reviews. Your feedback would be much appreciated and a link to the draft document is at the bottom of this slide.

We try to be transparent about the ideal steps and more realistic steps a reproducibility reviewer might take.

Ideal is if the reproducibility reviewer reads the abstract and DASA section, follows instructions in the README to reproduce the computations, and then writes the report.

More realistic however is: read abstract and DASA section, skim methods section, identify computer-generated figures and tables, follow all links in DASA section, start reproduction by trying to connect figures with the respective code or scripts, communicate with author, retry reproduction, improve the documentation of the computing environment, finish partial reproduction, write report with many recommendations what authors could improve, then be done.

The role is set out to be critical reviewer, but in part also educator. Reviewers should encourage and motivate, but not fix any problems. We accept irreproducibility if the author is not interested in making it work.

Regarding reproducibility reviewer skills, we see familiarity with package managers and virtual environments as crucial; while anyone should be able to reproduce any workflow with enough time and good enough documentation, we recommend to ask the author for help if one cannot get started within minutes, and stop if getting a workflow to run takes longer than one hour.

The section ends with a table of “Do’s and Don’ts” of reproducibility reviewer behaviour. Here are some examples:�Do not fix bugs.

Do encourage and share your knowledge by giving feedback.

Do document provided feedback in the reproducibility report to ensure traceability of your scientific contribution as a reviewer.

Do acknowledge the challenges that individual authors might face, for example structural, institutional or personal disadvantages.�And finall,y the hardest “don’t” in my experience: ”no rummaging”. It is really hard not to get excited and dig deeper all the time feeling “I am so close to get this to work” !

41 of 53

🙌

41

How to put your community on a path towards�more reproducibility in 5 easy hard steps

Build a team of enthusiasts (workshop, social events)
Assess the current state and raise awareness (workshop, paper)
Institutional support (🙏 AGILE Council 🙏 + committee chairs)
Positive encouragement (no reproduction != bad science)
Keep at it!

In just three years, we were able to plant the seed of cultural and structural change (if I myself may say so). So here is how to put your community on a path towards more reproducibility in 5 easy steps:

Build a team of enthusiasts, for example by organising a workshop
Assess the current state and talk about it - this is how people will realise you are serious; in our case it was pre-conference workshops and a paper with a strong message
Try to get institutional support - in our case the support of the AGILE Council and the conference committee chairs made everything possible
Try to get the community on your side by staying positive and inclusive - irreproducibility is usually not a sign for bad science, but for a lack of knowledge and incentives, which you can try to create
Keep at it, cultural change takes time - putting reproducibility on the map is an achievement in itself, and every little step counts; providing feedback as part of the reproducibility reviews will, one author at a time, educate your community.

42 of 53

The guidelines for data

“What if…” and Examples (not shown)

42

https://doi.org/10.17605/OSF.IO/CB7Z8

43 of 53

The guidelines�for workflows

Examples (not shown)

43

https://doi.org/10.17605/OSF.IO/CB7Z8

44 of 53

The guidelines for reproducibility reviewers (WIP)

Examples for “Do’s and Don’ts”:

Do shift burden to author
Do encourage and set examples
Do not accept private data sharing
Document your work in report (impact)
Be kind (career stage, knowledge, privileges)
No rummaging

44

https://docs.google.com/document/d/1Kc-ToUVcrdsq6aB8Qy2J_rIluFwDniv6GHGtZuPvIEo/edit#

Here are some examples for things to do and not to do.

We want to shift the main burden to the author;

Reviewers are very welcome to encourage and share their knowledge;

Reviewers should not accept private sharing of data without very good reason;

The reproducibility review is a contribution to science, that is why the recommendations and feedback should be documented and is published in a document with a DOI.

The reviewers are encouraged to acknowledge the challenges that individual authors might face, for example disadvantages they have, and be kind.�So far, that was not a problem at all, of course, but it hopefully encourages more authors to be open and transparent.�

The hardest task in my experience is ”no rummaging”: it is really hard not to get excited and dig deeper all the time feeling “I am so close” !

45 of 53

Structural challenges

Metrics for acknowledging/measuring impact in science are broken (impact factor, ..) and they lead to publication bias, HARKing, p-Hacking, intransparency and lack of reproducibility

Leiden Manifesto: http://www.leidenmanifesto.org�DORA: https://sfdora.org �Vienna Principles: https://viennaprinciples.org

Acknowledging data and software as valuable products of research (instead of shoehorning software into papers)

https://doi.org/10.31234/osf.io/p6e9c

https://simplystatistics.org/2017/07/26/announcing-the-tidypvals-package/

http://doi.org/10.5334/dsj-2020-003

46 of 53

https://doi.org/10.1038/467753a

“I am a professional software engineer and I want to share a trade secret with scientists: most professional computer software isn't very good. The code inside your laptop, television, phone or car is often badly documented, inconsistent and poorly tested.”�“Why does this matter to science? Because to turn raw data into published research papers often requires a little programming, which means that most scientists write software. And you scientists generally think the code you write is poor. It doesn't contain good comments, have sensible variable names or proper indentation. It breaks if you introduce badly formatted data, and you need to edit the output by hand to get the columns to line up. It includes a routine written by a graduate student which you never completely understood, and so on. Sound familiar? Well, those things don't matter.”

No excuse is good enough!

What can we do to make the software better?

47 of 53

Traditional and modern scientists

T

https://en.wikipedia.org/wiki/T-shaped_skills

https://doi.org/10.1007/s10816-015-9272-9

https://jakevdp.github.io/blog/2014/08/22/hacking-academia/

https://www.sciencemag.org/careers/2013/05/when-all-science-becomes-data-science

https://escience.washington.edu/community-level-data-science-and-its-spheres-of-influence-beyond-novelty-squared/

Π

Broad knowledge: across disciplines�collaborate with other experts, apply outside of own field

Deep knowledge: expertise and skills within a single field

Computer & method skills�statistics, reproducibility, programming, data science

https://en.wikipedia.org/wiki/T-shaped_skills

The concept of T-shaped skills, or T-shaped persons is a metaphor used in job recruitment to describe the abilities of persons in the workforce. The vertical bar on the letter T represents the depth of related skills and expertise in a single field, whereas the horizontal bar is the ability to collaborate across disciplines with experts in other areas and to apply knowledge in areas of expertise other than one's own.

https://escience.washington.edu/community-level-data-science-and-its-spheres-of-influence-beyond-novelty-squared/

The idealized characterization of data science in academia is also represented in the idea of shifting from the traditional T-shaped scientists, who have deep expertise in a single domain, to Π (Pi)-shaped scientists with deep expertise in both a domain and methodological science (as coined by Alex Szalay and discussed here and here). [...]

We encountered lots of data science, emerging across a network of interactions across a multitude of differently shaped scientists, from T to Γ (Gamma) to Π to even Μ (Mu) -shaped! Individuals mostly did not see themselves as particularly Π-shaped, a category many reserved for only a very select few that had accomplished an exceptional level of expertise and contribution to multiple fields.

By participating in the RSE community, you are on your path to become a modern pi-shaped scientists!

However, Pi-shaped people only get so far on their own! At some level of complexity, professionalisation is important.

48 of 53

≠

Images: https://pxhere.com/en/photo/477458 https://pxhere.com/en/photo/1087259 https://pxhere.com/en/photo/703106 https://pxhere.com/en/photo/103038

Professionalisation

49 of 53

Motivation

Back to 2010 The Software Sustainability Institute (SSI, UK) run a study (1000 randomly chosen researchers) …

“It's impossible to conduct�research without software,�say 7 out of 10 UK�researchers”

https://www.software.ac.uk/blog/2014-12-04-its-impossible-conduct-research-without-software-say-7-out-10-uk-researchers

50 of 53

Motivation

A study of Nature papers from Jan-March 2016 reveals that

“32 of the 40 papers examined mention software, and the 32 papers contain 211 mentions of distinct pieces of software, for an average of 6.5 mentions per paper.”

[2] Nangia, Udit; Katz, Daniel S. (2017): Understanding Software in Research: Initial Results from Examining Nature and a Call for Collaboration. doi:10.1109/eScience.2017.78

51 of 53

Image: Event Horizon Telescope

52 of 53

DEVELOPERS

CONDITIONS FOR

LEAD TO

EDUCATION OF

RESEARCHERS USING

53 of 53

”Software is 95% human and only�5% code” *

* Eric Albers, CCC2019, https://media.ccc.de/v/thms-49-ber-die-nachhaltigkeit-von-software �Bilder © H. Seibold, S. Janosch, OSD2019

RSEng = create research software

RSEs = people behind research software

RSEs ≠ IT !!!

Researcher uses scripts for data analysis and needs working stable software for her work. She learns what is necessary to achieve her research goals.

Reproducibility guru dives deeply into manifold software and tools to make his research reproducible and develops his own software in a sustainable way.

Person for tough problems knows how to solve all kinds of computer-related issues; he was not hired for that, but enjoys to help and spends time to get to the bottom of other people’s challenges.

Geek writes software as part of her research project and would like to code more, but must keep an eye on her career in science and needs to write papers.

Software developer was hired to implement software for a research project and contributes to large collaborative software projects to realise the next generation of digital infrastructure for science.

RSEng = Forschungssoftware bauen

RSEs = die Menschen hinter der Forschungssoftware

RSEs ≠ IT !!!

Die Forscherin benötigt Skripte für Datenanalyse und funktionierende Software für ihre Forschung, sie bringt sich dafür selbst bei, was nötig ist.

Reproducibility Guru arbeitet sich in neueste Softwarewerkzeuge ein um seine Forschung reproduzierbar zu machen und Software nachhaltig zu entwickeln.

Die Gute Seele für kniffliges im Institut die weiß wie alle möglichen Computerprobleme gelöst werden können, wurde eigentlich für andere Dinge (Forschung?!) eingestellt aber ist zu hilfsbereit und hat Freude daran anderen mit ihrer Forschungssoftware zu helfen.

Der Geek schreibt Software als Teil seines Forschungsprojekts und würde gerne mehr coden, aber muss an die Karriere denken und Artikel schreiben.

Die Softwareentwicklein wurde eingestellt um Software für ein Forschungsprojekt zu entwickeln und trägt in kollaborativen Projekten zur Entwicklung der nächsten Generation wissenschaftlicher digitaler Infrastruktur bei.

Alle diese Rollen tragen maßgeblich zur Entwicklung von Software für Forschung bei, aber bis auf die letzten zwei würde keine sich selbst als Research Software Engineer bezeichnen, und _keine_ dieser Personen trägt heute die Buschstaben “RSE” in der Aufgabenbeschreibung oder gar in der Stellenbezeichnung.

And mixtures of all these roles exit!