1 of 37

Robust Research

A practical guide

Verena Heise

Meta-Psychology Symposium

7 June 2018

2 of 37

What can I do?

3 of 37

https://osf.io/tvyxz/wiki/home/

Open Science

4 of 37

https://osf.io/tvyxz/wiki/home/

Open Science

5 of 37

Open Data

Publish data in repositories: https://osf.io/ https://zenodo.org/
Publish data description papers: Scientific Data Journal https://www.nature.com/sdata/

Neuroimaging:

Organise data according to BIDS format: http://bids.neuroimaging.io/
Publish summary stats results on https://neurovault.org/

6 of 37

Open Data

7 of 37

https://osf.io/tvyxz/wiki/home/

Open Science

8 of 37

Open Materials

Publish the analysis scripts (ALL of them and make them readable ☺)
Describe (and share) experimental setups, equipment, etc.

9 of 37

https://osf.io/tvyxz/wiki/home/

Open Science

10 of 37

Open Science

Open Reporting

11 of 37

Open Reporting

Publish ALL the analyses you did (pre-registered and exploratory)
Publish ALL results (not just “significant” ones)
Publish according to best practice guidelines

12 of 37

Best practice guidelines

http://www.equator-network.org/

13 of 37

Open Reporting

Publish ALL the analyses you did (pre-registered and exploratory)
Publish ALL results (not just “significant” ones)
Publish according to best practice guidelines
Use preprints
Publish Open Access

14 of 37

Good Scientific Practice

15 of 37

Ask the right question

Chalmers et al., The Lancet 2014, https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62229-1/fulltext

16 of 37

Good Scientific Practice

17 of 37

Do you need to collect new data?

UK Biobank: http://www.ukbiobank.ac.uk/
Human Connectome Projects: https://www.humanconnectome.org/
Many other collaborations, e.g. in disease-specific areas

18 of 37

Design your study properly

http://www.equator-network.org/

19 of 37

Statistical power

20 of 37

General statistical power: http://www.gpower.hhu.de/en.html
fMRI power tools: http://neuropowertools.org/ http://fmripower.org/

Sample size calculators

21 of 37

Effect size = Cohen’s d of 0.5 (medium effect)
Statistical power to find effect = 90%
alpha = 0.05
One sample one-tailed t test
36 participants
Independent groups two-tailed t test
86 participants per group

Sample size calculation - example

22 of 37

Reproducible workflows

Keep an electronic labbook
Script as much as you can
Helpful tools: R Studio: https://www.rstudio.com/ R Markdown: https://rmarkdown.rstudio.com/ Python: https://www.python.org/ Jupyter Notebook: https://jupyter.org/

23 of 37

Reproducible measures

Validity (Can I get the right answer?)
Reliability (Can I get the same answer twice?)

Reliable

Not Valid

Valid

Not reliable

Not valid

Both reliable and valid

24 of 37

Reproducible measures

How reliable and valid are your tests?

Can you compare with gold standard or well-established tests?

Are you doing any quality control of your tools (experimental setup, acquired data, etc.)
Analysis pipelines

Use well-established tools
Follow good programming practice

Test your code using simulations

25 of 37

Robust Research - summary

Open Science

Data
Materials
Reporting (and pre-registration)

Good Scientific Practice

Relevant research question
Statistical power
Reproducible workflows
Reproducible measures (validity, reliability, QC procedures)

26 of 37

https://en.wikipedia.org/wiki/The_Scream

27 of 37

What can we do?

28 of 37

The Oxford experience

Started Robust Research Initiative in September 2017
Mainly early career researchers
Disciplines: experimental psychology, biomedical sciences (preclinical to clinical), social sciences (archaeology, anthropology, economics), bioethics

29 of 37

Robust Research

Research

Ethics

Infrastructure

Collaborative research

Helpdesks

Quality control

Innovation

Education

Undergraduate/ Graduate

Postdocs/ staff/ PIs

External training

Incentives

Metrics

Hiring/ promotion criteria

Award system

Policy

Funders

Publishers

National/ International networks

Government

30 of 37

Education

Journal clubs
Seminar series
Workshops
Summer school in collaboration with Berlin: https://www.bihealth.org/de/aktuell/berlin-oxford-summer-school/
Provide speakers for lab meetings
Undergraduate teaching
Lecture series/ workshops for taught MSc programmes
Doctoral training centres
Skills training for medical sciences division

31 of 37

Why do we care?

32 of 37

https://osf.io/tvyxz/wiki/home/

Open Science

33 of 37

Translation

34 of 37

Ethics

Waste of animal lives
Waste of patients’ time, hope and lives
Waste of money (up to 85% of research funding, Chalmers and Glasziou, Lancet 2009 https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(09)60329-9)

35 of 37

Trust in evidence

36 of 37

37 of 37

Questions or comments?

verena.heise@ndph.ox.ac.uk