Reproducibility, Transparent Methods, and Pre-Registration

​

Reproducibility and Transparent Methods

https://youtu.be/A6ASIZPAtbo

2

Reproducibility & Evidence Synthesis

​

Evidence synthesis methods should be reproducible

​

​

​

Confidence in science

​

3

DEFINITIONS: REPRODUCIBILITY

“the ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator."

​

​

4

Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.

​

​

Evidence that you weren’t cherry picking studies

Other scholars can check your work

DEFINITIONS: REPLICABILITY

“the ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.”

​

​

​

​

5

Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.

​

​

living systematic review (LSR)

• Re-running/updating your own searches
• Replication & Extension

Research methods should be

transparent & reproducible

6

PsycINFO (Ovid)

1. exp infant development/
2. (babies OR baby* OR infanc* OR infant*).ti,ab,id
3. Exp Screening/
4. (Screen* OR test* OR evaluation* OR diagnostic* OR detection* OR identification*).tm,ti,ab,id
5. (1 OR 2) AND (3 OR 4)

n=31402

March 3, 2025

n=31402

n=31402

Many databases and aggregators are doing exactly what we are telling them to do

Search for these words in these fields

​

(babies OR baby* OR infanc* OR infant*).ti

TI (babies OR baby* OR infanc* OR infant*)

7

Many databases and aggregators are doing exactly what we are telling them to do

Search for these words in these fields

(babies OR baby* OR infanc* OR infant*).ti

TI (babies OR baby* OR infanc* OR infant*)

8

Barrick, K., & Riegelman, A. (2021). Phrasing in reproducible search methodology: The consequences of straight and curly quotation marks. College & Research Libraries, 82(7), 978. doi:10.5860/crl.82.7.978

Some don't

Forcing Exact Search

9

Forcing Exact Search

10

Forcing Exact Search

11

Peer review example

Peer reviewers may want to test your exact search strategy to try to examine why a potentially relevant study was not included in your review.

• If a search was last conducted on December 1, 2022, a peer reviewer would have a clear answer for why a March 2023 published study was not included in a review.

12

Problems outside of our control that can affect the reproducibility of the results

Database or search engine…

Syntax changes

Algorithm changes

Retrospective cataloging

Subject heading re-assignments

Adding or dropping journals (e.g., delisting journals for failing to meet their standards)

13

Problems outside of our control that can affect the reproducibility of the results

Database or search engine…

Default settings that vary between institutions

No longer exists

The person attempting to reproduce the search strategy lacks access to proprietary databases or platforms

14

Some people think that pre-registration helps reproducibility

Proxy for quality

Pre-Registration

• Stating your research plan before gathering data
• Exploratory vs Confirmatory

​

16

Garden of forking paths

Rigorous methods driven as opposed to results driven (e.g., cherry picking)

17

Preregistration

≠

reproducibility

Registering a protocol could thwart questionable research practices

​

​

18

Transparency

≠

reproducibility

No reproducibility w/o transparency

19

Evidence synthesis and how it relates to confidence in science

Evidence-based decisions as opposed to a hunch or one-off studies

Informed choices based on all of the available evidence

21

🙌

Garbage in, Garbage out

Reviews are only as good as the included studies

​

Reviewers can conduct critical appraisal and analysis of included studies

​

​

But…

22

Garbage in, Garbage out

Reviews are only as good as the included studies

​

Reviewers can conduct critical appraisal and analysis of included studies

​

​

But…

23

​

High risk of bias

​

​

​

Low risk of bias

​

​

Confidence in Science

24

Source: https://retractionwatch.com/2022/11/09/pain-researchers-lose-three-papers-after-cochrane-group-questioned-data/

7

Monticone's research scrutinized

Systematic review team questioned reliability of the research all w/ one author in common-- Marco Monticone.

Cause for concern:

• Statistically impossible data
• "Duplicated or very similar data from other studies" that were wrongly reported as separate clinical trials.
• "extremely large effect sizes"
• "low or zero attrition" in studies

25

Scientific misconduct

26

Sometimes flawed research is unintentional

Fields like Psychology are starting to pay more attention to these things

See Questionable Research Practices (QRPs)

27

Questionable Research Practices (QRPs)

• p-Hacking
• HARKing
• Underpowered Studies
• Publication Bias / File Drawer Problem

​

28

Bias Against the Null / File Drawer Problem

• Tenure system not rewarding replications, data sharing, reporting
• Journals reluctant to publish null results
• Authors putting null results in the file drawer

29

How does this apply to evidence synthesis?

• The gold standard for many review types is to seek unpublished studies via grey literature, and therefore null and negative findings could be represented in a review sample

​

Chapter 7. Confidence in Science

Evidence synthesis is

• “A robust means for gaining confidence in scientific knowledge over time…”
• “A method for “assessing the reliability and validity of bodies of research”

30

doi:10.17226/25303

Questions?

31