1 of 24

Improving reproducibility of geospatial conference papers

Lessons learned from a first implementation�of reproducibility reviews

Daniel Nüst (AGILE 2020 Reproducibility Chair), Frank O. Ostermann,�Carlos Granell, Alexander Kmoch (all Reproducibility Committee)�

https://doi.org/10.7557/5.5601

Slides CC-BY 4.0

2 of 24

https://agile-online.org/

AGILE council | annual conference | PhD schools | initiatives

GIScience teaching/research @ European research agendas

2

3 of 24

https://reproducible-agile.github.io/

2017, ‘18 & ‘19: Workshops on reproducibility�2019: Reproducible publications at AGILE conferences (initiative)�2020: First AGILE reproducibility review

3

Two workshops in 2017 and 2018 and a piece of meta-research into the reproducibility of papers published at the AGILE conference were the basis for a small group from the AGILE community to apply for an initiative to develop guidelines for reproducible publications. The AGILE council granted support for the initiative, and the guidelines were published shortly before the conference in 2019.�In 2020, the guidelines were for the first time recommended in the call for papers, and for the first time a reproducibility committee was put in place to evaluate the reproducibility of accepted papers. Find out more at the Reproducible AGILE website.

Today, I will introduce the Reproducible Paper Guidelines. Then, I’ll talk about the reproducibility review process at the last conference in detail and try to summarise the results and lessons learned in a way that makes them useful to other communities.

4 of 24

AGILE Reproducible Paper Guidelines:�Contents & First Revision

4

5 of 24

AGILE Reproducible Paper Guidelines 🇬🇧 🇪🇸

https://doi.org/10.17605/OSF.IO/CB7Z8

Created by AGILE Initiative in 2019, see report at https://osf.io/hupxr/

Transparency & Reproducibility�GIScience�https://osf.io/phmce/wiki/home/

Promotion�Acknowledge spectrum

5

The AGILE Reproducible Paper Guidelines are the first compilation of tips and tools how to increase transparency and reproducibility in GIScience papers.

They are complemented with a Wiki with examples for typical GIScience datasets and workflows.

The guidelines’ intention is to promote positive examples of reproducible computational research and to acknowledge the full spectrum of reproducibility as well as the transition towards establishment of reproducible research practices.

The only requirement of the guidelines is that each paper must include a Data and Software Availability sub-section, or “DASA section” for short. It is absolutely possible not to provide access to data and code and to still comply with the guidelines at the same time, as long as there are good reasons given.

6 of 24

The guidelines

Author guidelines�Data in Research Papers�Computational workflows �in Research Papers�Pre-submission checklist�Writing DASA section

Rationale/Motivation/Vision

Reviewer guidelines

Reproducibility reviewer guidelines (WIP)

6

https://doi.org/10.17605/OSF.IO/CB7Z8

7 of 24

The guidelines for

reproducibility reviewers (WIP)

Ideal vs. realistic

Role

Skills

Do’s & dont’s

7

https://docs.google.com/document/d/1Kc-ToUVcrdsq6aB8Qy2J_rIluFwDniv6GHGtZuPvIEo/edit#

[4:00]��I will skip the details of the author guidelines, but I’d like to take a look at the brand new reproducibility reviewer guidelines, because they really set the tone and scope of how we do reproducibility reviews. Your feedback would be much appreciated and a link to the draft document is at the bottom of this slide.

We try to be transparent about the ideal steps and more realistic steps a reproducibility reviewer might take.

Ideal is if the reproducibility reviewer reads the abstract and DASA section, follows instructions in the README to reproduce the computations, and then writes the report.

More realistic however is: read abstract and DASA section, skim methods section, identify computer-generated figures and tables, follow all links in DASA section, start reproduction by trying to connect figures with the respective code or scripts, communicate with author, retry reproduction, improve the documentation of the computing environment, finish partial reproduction, write report with many recommendations what authors could improve, then be done.

The role is set out to be critical reviewer, but in part also educator. Reviewers should encourage and motivate, but not fix any problems. We accept irreproducibility if the author is not interested in making it work.

Regarding reproducibility reviewer skills, we see familiarity with package managers and virtual environments as crucial; while anyone should be able to reproduce any workflow with enough time and good enough documentation, we recommend to ask the author for help if one cannot get started within minutes, and stop if getting a workflow to run takes longer than one hour.

The section ends with a table of “Do’s and Don’ts” of reproducibility reviewer behaviour. Here are some examples:�Do not fix bugs.

Do encourage and share your knowledge by giving feedback.

Do document provided feedback in the reproducibility report to ensure traceability of your scientific contribution as a reviewer.

Do acknowledge the challenges that individual authors might face, for example structural, institutional or personal disadvantages.�And finall,y the hardest “don’t” in my experience: ”no rummaging”. It is really hard not to get excited and dig deeper all the time feeling “I am so close to get this to work” !

8 of 24

Reproducibility Review at�AGILE Conference 2020

8

9 of 24

Review process

Proceedings:�https://www.agile-giscience-series.net/review_process.html

Process documentation:�https://osf.io/7rjpe/

9

Reproducibility review after accept/reject decisions, triggered by regular reviewer

Reproducibility review & communication

Community conference & coronavirus

Badges on proceedings page

Presentation at conference

First a few words about the process. You can find the details in the linked document.

Let’s start with the order: the reproducibility review happened after the authors were notified about the acceptance or rejection. Ideally, we wanted to review only papers that were identified by the scientific reviewers, but we found that that was not reliable so we briefly checked all papers ourselves. Also, none of the regular reviewers could be intrigued to attempt a reproduction.

This reduced the number of papers and allowed us to communicate directly with the authors during the reproducibility reviews.

The process was not smooth, because AGILE is a community-led conference managed by volunteers, and the authors are the ones who prepare the camera-ready copies of the articles. So there is a lot of moving parts with no central control. On top of that, the conference switched the publisher and the Coronavirus destroyed all schedules.

Eventually, the papers were published. The publisher added badges on the landing pages of the proceedings, which link to the reproducibility reports, as you can see on the right hand side. The papers themselves do not reference the reports.

We originally planned to have short presentations of selected reproducible papers in the final conference session, where candidates for the the best paper award are presented and voted live by the audience. Unfortunately, the in-person event was cancelled, so we could not have a showcase of the first batch of reproducible papers as we would have liked to.�

10 of 24

Reproducibility review results

6 reproducibility reports published

16 not possible/not attempted�(5 of which after communication with authors):

no starting point in the paper
documentation insufficient for third party
sensitive/confidential/commercial data
proprietary software
software paper
(conceptual papers)

10

https://osf.io/6k5fh/

Six reproductions were partially successful; “Partially”, because not all details of the computational results, for example not all figures, were reproducible.

16 papers were assessed as impossible to reproduce early on, for different reasons. Here are barriers we faced:

If authors provided no starting point, that is no link to data or code at all, we did not attempt a reproduction even if there clearly was a computational workflow; we neither contacted the authors in these cases;

Often documentation was not detailed enough for a third party to reproduce a workflow, for example generic dataset citations or manual steps, especially for data download, or anonymisation that was not removed in a timely manner;

Some papers use sensitive data on human subjects, or confidential or commercial data, which we did not have access to;

Some papers relied on proprietary software;

One paper was simply describing a piece of software, which is fine, but didn’t give us anything to evaluate;

similarly, in previous years of the conference, we observed purely conceptual papers, which of course are also not computationally reproducible.

In all of these cases, authors could have documented the reason themselves, for example in the DASA section, but few did.

11 of 24

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/ZTC7M

https://doi.org/10.17605/OSF.IO/7XRQG

11

12 of 24

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/XS5YR

https://doi.org/10.17605/OSF.IO/SUWPJ

12

13 of 24

Reproducibility review reports

https://doi.org/10.17605/OSF.IO/7TWR2

https://doi.org/10.17605/OSF.IO/5SVMT

13

On the left, we have a paper that used an anonymous supplement, but needed some back and forth to get the workflow right; the authors provided a textual definition of the computing environment, which the reproducibility reviewer turned into a standardised format; eventually the plots were pretty close to originals from the paper, though you can see here that the axes labels are illegible in the reproductions.

On the right hand side, we have a paper with data and code on OSF; with a lot of trial and error with some author help, eventually we got a subset of the workflow running, because the full analysis was much too large; the recreated figures were close enough to the original to trust everything is there, though there was no perfect match (for example, different figure titles).

Please take a look at these reports and let me know if you think they are useful for authors, students, readers, or reviewers, and how we can improve them. So what do we do with these reports?

14 of 24

https://codecheck.org.uk/register/

14

Independent execution of computations underlying research articles.

15 of 24

Findings

Overall

Saw full spectrum of reproducibility
Compared to previous years’ submissions, the guidelines and increased community awareness markedly improved reproducibility
⅚ reproduced papers have DASA; all embrace guidelines
Reproducibility reports with many recommendations for improvement, well received by authors, even included in revision before publication > reward!
Good practices spread slowly
Process

15

Read full report at https://osf.io/7rjpe/

This is the first reproducibility review at an AGILE conference, and as you can expect from a first time, we deviated quite a lot from the original plan. Here is what we learned:

We saw the full spectrum of reproducibility, which encourages us to continue with our current approach.

We saw that compared to previous years’ submissions, the guidelines did effectively increase community awareness and improved reproducibility. Of course we also looked harder than before, but we are still very happy with the overall outcome of the reproductions.

Almost all reproduced papers have the DASA section, and all follow them at least in part. That’s a great sign that even recommended guidelines have impact.

The reproducibility reports have a lot of details with recommendations how to improve workflows. Some authors picked these up pretty quickly and even improved their material in time with the publication of the articles, which I think is a great reward for the reproducibility reviewers.

Also, good practices spread slowly because people need to change their (even daily) habits; this also led to us being less strict than originally anticipated, but that’s a good flexibility to let the community of authors influence the pace of change, too. Slow change can be a good thing.

And finally, the process cannot be established at once, but needs to be improved over time, especially the coordination with other conference committees and the publisher. For example, we plan to not allow authors to object the publication of the reproducibility report anymore. Since only successful reproductions will be published anyway, we don’t see a need for this additional step. The need to credit the reviewer also outweights possible author’s concerns here.

----------

Image: https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Linear_visible_spectrum.svg/1024px-Linear_visible_spectrum.svg.png

16 of 24

Findings

Challenges for reproducibility reviewer:

Inconsistencies (identifiers, links) between paper and code
Lack of connections between artefacts (code <> figure)
Workspaces layout: no documentation, absolute paths
Unknown runtime and no demo subsets of data
No guidance on efforts and stop points

�All efforts beyond mere workflow execution

16

Read full report at https://osf.io/7rjpe/

These are the most common challenges that the reproducibility reviewers faced:

We found inconsistencies between the versions and identifiers used in the data deposit and the manuscript. That is one of the main reasons why proper depositing (i.e., with a DOI) of data and code are so important.

On top of that, a lack of connections between figures in the paper and documentation/code really slowed reviewers down. If you are an author, please use the figure numbers in your README and file names. Ideally, mention the command or script file in the figure description, too.

Workspace layout and methods often were not accessible for third parties: no clear README file, usage of absolute paths; these are things that the guidelines do cover, but authors seem to be unaware how tedious these are to figure out.

Another challenge is undocumented run time, so it can happen that you start a process and after 24 hours it tells you there is 10% progress; authors need to be aware of this and provide subsets for quicker evaluation of methodologies.

This is related to the challenge that due to the lack of experiences, we did not provide any guidance on how much time reviewers should spend before stopping. I estimate that we spent 4 to 10 person hours per review this year, not including running the actual computations, which is not feasible in the long term.

To summarise, I would say that these are all problems outside of the mere execution of the workflow. This is where authors can do a better job to reduce the time needed to understand their method by preparing more reproducible workflows. Trying to attempt a reproduction yourself could be a great way to increase mutual understanding.

----------

Image: https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Linear_visible_spectrum.svg/1024px-Linear_visible_spectrum.svg.png

17 of 24

🙌

17

How to put your community on a path towards�more reproducibility in 5 easy hard steps

Build a team of enthusiasts (workshop, social events)
Assess the current state and raise awareness (workshop, paper)
Institutional support (🙏 AGILE Council 🙏 + committee chairs)
Positive encouragement (no reproduction != bad science)
Keep at it!

In just three years, we were able to plant the seed of cultural and structural change (if I myself may say so). So here is how to put your community on a path towards more reproducibility in 5 easy steps:

Build a team of enthusiasts, for example by organising a workshop
Assess the current state and talk about it - this is how people will realise you are serious; in our case it was pre-conference workshops and a paper with a strong message
Try to get institutional support - in our case the support of the AGILE Council and the conference committee chairs made everything possible
Try to get the community on your side by staying positive and inclusive - irreproducibility is usually not a sign for bad science, but for a lack of knowledge and incentives, which you can try to create
Keep at it, cultural change takes time - putting reproducibility on the map is an achievement in itself, and every little step counts; providing feedback as part of the reproducibility reviews will, one author at a time, educate your community.

18 of 24

Next steps

Do it again in 2021 🎉

Revise guidelines 🛠️ 🇮🇹 🇫🇷 🇨🇳

Grow reproducibility reviewer team� ECRs, credit @ ORCID, skills

Continue research 🕵️� Ostermann, F., Nüst, D., Granell, C., Hofer, B., & Konkol, M. (2020).� Reproducible Research and GIScience: an evaluation using � GIScience conference papers.� EarthArXiv. https://doi.org/10.31223/x5zk5v

18

Continue community engagement towards opening scholarship� Scope� Requirements� Acceptance condition?� � Open review if tenured� Format-free submission� CRediT� Phase out when standard practice...

I’d like finish with a short outlook on the next steps.

Most importantly, we are doing it again next year and try to learn from our experiences.

Furthermore, the guidelines will be improved. Most likely we will simplify them and move details to other places. I’m keen on having the guidelines translated to make them more accessible, and hope they find adoption beyond the conference.

We need to grow reproducibility reviewer team, which I think is a great opportunity to involve early career researchers in peer review. More people means more diverse skill sets to better match the given manuscripts.�I hope that we will be able to give reproducibility (and regular) reviewers public credit soon, for example by putting reviewer activity into ORCID profiles.

We continue to research the state of reproducibility in GIScience and have just published a preprint on another conference series, the GIScience conference. The results are equally sobering as for the AGILE conference three years ago.

And, of course, the continued interaction with the community is key to increase reproducibility further. We will debate the scope of the review and when to increase requirements, maybe even require reproductions for acceptance?�In the long term, I personally hope the awareness on reproducibility is a starting point for more open practices in general, for example requiring tenured reviewers to disclose their identity, accept first submissions without formatting, or using the CRediT taxonomy for author contributions.�And most importantly, I’d be completely fine to phase out the special reproducibility review, when reproducible papers have become standard practice.

19 of 24

Thank you!

I look forward to your questions!�@nordhomen | d.n@wwu.de�

Slides: https://doi.org/10.7557/5.5601

�Reproducibility Committee 2020 + Initiative�Daniel Nüst (University of Münster, GER)�Frank Ostermann (University of Twente, NEL)�Carlos Granell (Universitat of Jaume I, ESP)�Alexander Kmoch (University of Tartu, EST)�Barbara Hofer (University of Salzburg, AUT)�Rusne Sileryte (TU Delft)�Markus Konkol (University of Twente, NEL)

https://reproducible-agile.github.io/

Word-stem cloud of all AGILE 2020 submissions (full/short/poster)

19

Slides published under CC BY 4.0

20 of 24

Bonus slides

20

21 of 24

The guidelines for data

“What if…” and Examples (not shown)

21

https://doi.org/10.17605/OSF.IO/CB7Z8

22 of 24

The guidelines�for workflows

Examples (not shown)

22

https://doi.org/10.17605/OSF.IO/CB7Z8

23 of 24

The guidelines for reproducibility reviewers (WIP)

Examples for “Do’s and Don’ts”:

Do shift burden to author
Do encourage and set examples
Do not accept private data sharing
Document your work in report (impact)
Be kind (career stage, knowledge, privileges)
No rummaging

23

https://docs.google.com/document/d/1Kc-ToUVcrdsq6aB8Qy2J_rIluFwDniv6GHGtZuPvIEo/edit#

Here are some examples for things to do and not to do.

We want to shift the main burden to the author;

Reviewers are very welcome to encourage and share their knowledge;

Reviewers should not accept private sharing of data without very good reason;

The reproducibility review is a contribution to science, that is why the recommendations and feedback should be documented and is published in a document with a DOI.

The reviewers are encouraged to acknowledge the challenges that individual authors might face, for example disadvantages they have, and be kind.�So far, that was not a problem at all, of course, but it hopefully encourages more authors to be open and transparent.�

The hardest task in my experience is ”no rummaging”: it is really hard not to get excited and dig deeper all the time feeling “I am so close” !

24 of 24

0% of rejected papers have a DASA section (correlation, not cause)�48% of accepted full papers have DASA section

24

Reproducible research and GIScience: an evaluation using AGILE conference papers�https://doi.org/10.7717/peerj.5072

Let’s take a look at some numbers:

Out of 23 accepted full papers, 11 have a DASA section, which is about 48%.

None of the rejected papers had a DASA section, but that is likely a correlation and not a causation of course, though one main argument for higher transparency and reproducibility is indeed that the method can be presented in a more convincing way to reviewers.

I think this is a really great achievement for the whole community and I want to thank all authors of these 11 manuscripts for their open-mindedness and their engagement for open science.

Let’s not forget that in 2019

none of the papers had a DASA section, and our study from 2018 showed that none of the 32 best paper nominees (full and short papers) from 2010 to 2017 would have provided enough information to even attempt a reproduction.

So how many reproductions did we attempt this year?