SCIENTIFIC ECOSYSTEMS AND RESEARCH REPRODUCIBILITY
Marcus Munafò
Open Science Collaboration (2015). Science, 349.
Dreber et al. (2015). PNAS, 112, 15343-15347.
Prediction market on the outcomes of the Reproducibility Project: Psychology
Successful replications are shown in black, unsuccessful replications in red.
“Scientists may be in the business of laughing at their predecessors, but owing to an array of human mental dispositions, few realize that someone will laugh at their beliefs in the (disappointingly near) future”
Taleb (2007). Fooled by Randomness.
“Certain features of the working environment of science may have unexpected and potentially detrimental effects on the ethical dimensions of scientists’ work”
Martinson et al. (2005). Nature, 435, 737-738.
Neuroskeptic (2012). Perspect Psychol Sci, 7, 643-644.
(i) Limbo
(ii) Overselling
(iii) Post-hoc storytelling
(iv) P-value fishing
(v) Creative outliers
(vi) Plagiarism
(viii) Non-publication
(viii) Partial publication
(ix) Falsification
http://blogs.discovermagazine.com/neuroskeptic/2013/10/16/the-f-problem
Simmons et al. (2011). Psychol Sci, 22, 1359-1366.
Carp (2012). Neuroimage, 63, 289-300.
“…nearly as many unique analysis pipelines as there were studies in the sample…”
Incentive Structures
Bones (2012). Perspect Psychol Sci, 7, 307.
Incentive Structures
Button et al. (2013). Nat Rev Neurosci, 14, 365-376.
Macleod et al. (2015). PLOS Biol, 13, e1002301.
Studies from top-ranked UK institutions perform worse on reporting of measures to reduce the risk of bias than studies selected at random from PubMed…
Fanelli & Ioannidis (2013). PNAS, 5, e10271.
Ioannidis (2012). Perspect Psychol Sci, 7, 645-654.
Tajika et al. (2015). Br J Psychiatry, 207, 357-362.
“Among 83 articles recommending effective interventions, 40 had not been subject to any attempt at replication…”
Panagiotou & Ioannidis (2012). J Clin Epidemiol, 65, 740-747.
Greenberg (2009). Br Med J, 339, b2680.
Investigated citation network of papers addressing the belief that B amyloid, a protein accumulated in the brain in Alzheimer’s disease, is produced by and injures skeletal muscle of patients with inclusion body myositis.
Misemer et al. (2016). Trials, 17, 473.
Two positive trials, four neutral trials, two negative trials (stopped early for safety concerns).
Munafò et al. (2014), Nat Biotech, 32, 871-873.
Kaplan & Irvin (2015). PLoS One, 10, e0132382.
In 2000 the National Heart Lung, and Blood Institute required the registration of primary outcome on ClinicalTrials.gov for all their grant-funded activity
Kidwell et al. (2016). PLoS Biology, 14, e1002456.
Introduction of badges for open practices at Psychological Science followed by a steep increased in data sharing.
Munafò et al. (2017). Nat Hum Behav, 1, 0021.
Beyond Replication…
Cartoon: Matteo Farinella
Acknowledgements
@MarcusMunafo
@BristolTARG
http://www.bristol.ac.uk/expsych/research/brain/targ
http://www.bris.ac.uk/expsych/events/reproducibility2018/
Tobacco and Alcohol Research Group:
Angela Attwood Senior Lecturer
Anna Blackwell Postdoc
Laura Brocklebank Postdoc
Emily Crowe PhD Student
Perline Demange Research Assistant
Maddy Dyer PhD Student
Kayleigh Easey PhD Student
Andy Eastwood PhD Student
Meg Fluharty PhD Student
Elis Haan PhD Student
Eleanor Kennedy PhD Student
Jasmine Khouja PhD Student
Glenda Lassi Postdoc
Rebecca Lawn PhD Student
Jim Lumsden PhD Student
Olivia Maynard Lecturer
Sarah Peters PhD Student
Vicky Rice PhD Student
Laura Schellhas PhD Student
Hannah Sallis Postdoc
Carlos Sillero Research Assistant
Andy Skinner Postdoc
Giulia Sorrentino Research Assistant
Chris Stone Research Assistant
Steph Suddell PhD Student
Gemma Taylor Postdoc
Jorien Treur Postdoc
David Troy PhD Student
Robyn Wootton Postdoc