The nudge you need to preregister your next project

Dr. Zsuzsa Kaldy

Professor

UMass Boston, Early Minds Lab

Developmental & Brain Sciences Program

babies.umb.edu

ManyBabies Workshop, 2/28/2025

Before we begin

• Hungarian scientist, working in the US since 1998
• Grew up in communist Hungary, witnessed the autocratization of my country since 2010
• Current moment: Unprecedented ideology-driven, across-the-board attacks on scientific research and universities in the US
• With world-wide effects
• Databases
• Academic freedom
• Professional organizations, journals, collaborations, careers
• Science needs our support: whether or not you are not in the US, push your academic leaders to work toward safeguards in your country

​

The nudge you need to preregister your next project

Dr. Zsuzsa Kaldy

Professor

UMass Boston, Early Minds Lab

Developmental & Brain Sciences Program

babies.umb.edu

ManyBabies Workshop, 2/28/2025

How should we conduct science?

• Scientists should evaluate claims based on the evidence at hand (universalism),
• Evidence should be openly available for inspection (communality),
• We should not be motivated by self-interest (disinterestedness),
• Claims should be calibrated with the evidence presented (organized skepticism).

-> Open science is just good science (Tennant, 2018, YouTube talk)

Preregistration is just one part of this program

Merton, R. K. (1973). The Sociology of Science:

Theoretical and Empirical Investigations

The goals of this seminar

• Show you the benefits of preregistration, with a focus on experimental studies
• Present some ways you can get around the obstacles that you may have
• Give you some practical help with getting started

​

• What we won’t do:
• Get into some of the specifics, e.g. sample size estimation
• Get into other aspects of open science (need for replication, data sharing, changing incentive structures, open access,…)

My qualifications

• I have been studying the development of visual attention and working memory in infants and young children for 25+ years
• Leading the Early Minds Lab at UMass Boston since 2003
• Serving as a reviewer for grants and papers; associate editor
• Preregistering all our studies on OSF since 2016

The fundamentals of open science

• The reproducibility crisis

The fundamentals of open science

• Solutions for the reproducibility (or credibility) crisis

The fundamentals of open science

• The “garden of forking paths” (Borges, Ficciones, 1944) or “researcher degrees of freedom”
• Confirmation bias, file drawer effect, p-hacking
• HARKing (Hypothesizing After Results are Known)

Does preregistration solve all our problems?

• No, if you want to cheat, you can still do it!
• There are many additional practices that foster robustness and reproducibility
• Important arguments about statistical model-testing: we can’t fix scientific reasoning and theory development with statistics (Szollosi et al., 2020, TICS)

You know it’s important, but…

• You’ve just heard something that’s clearly important to do – how come not all studies are preregistered???
• Most people have heard these arguments, and they still don’t preregister their studies
• BUT: Journals are increasingly moving toward using preregistration as a standard
• Now it’s carrots, soon (?) it may be sticks
• And, likely, at some point, grant agencies will follow

What is a preregistration?

• The original idea: clinical studies, Randomized Controlled Trials
• Very generically: a plan for…
• how to conduct your study, and what are your predictions
• how to analyze your data and how to test your predictions
• Main criterion: You need to publicly share it before you begin data collection*
• The level of detail is currently not set in stone, but there are some guidelines

When can you be justified to not preregister your study?

• Qualitative data (but even then, you can preregister your coding scheme)
• Some extremely exploratory projects (but surely, you have some methods/analyses that you are planning to run…)
• Longitudinal studies (in fact, that’s not even an excuse, see Petersen et al., 2022, Inf Child Dev)
• Meta-analysis (see Moreau & Gamble, 2020, Psych Methods)
• I am interested in pre-registrations for studies that use existing data/secondary data.

​

• So basically… never!

Common excuses for not doing it

• Distinguishing confirmatory from exploratory study
• A version of this excuse: I have a multi-experiment study
• No excuse: just preregister each study separately, before you launch the next one

​

1. I am a real pioneer of my field! I don’t know what I will discover, and I don’t want to be shackled

​

Common excuses for not doing it

• Not really: you just need to write your Methods and Data analysis sections sooner
• It saves you a TON of time after you have collected your data

​

2. It seems like a lot of extra work

​

Common excuses for not doing it

• If you are a PI: you know that’s not an excuse!
• If you are a graduate student or postdoc…
• You need to convince your PI
• If you can’t, you need to look for outside support
• If you are an undergraduate student…
• Find a grad mentor to work with who can describe their lab’s open science practices
• Culture and peer pressure are very strong forces

3. Nobody in my lab or program is doing it

​

Common excuses for not doing it

• This was my excuse for a long time!
• But this is just bad project management: if you are smart enough to plan a cutting-edge study, you need to be smart enough to plan your work a month ahead, too
• You can submit your preregistration after data collection has started if you have not looked at the results - in any sense of the word
• But honestly, just be better about planning!

​

4. I don’t have time, I have participants scheduled for next Tuesday

​

Common excuses for not doing it

• Ethical considerations
• You have total control over the access levels in OSF: private/public – if you are worried, you can keep it private until you published your paper
• A huge advantage of preregistration: you can unassailably document when you have started a project

​

5. I am afraid I am going to be scooped

​

Practical matters

• Planning
• It takes at least a month to do it right, more if it’s your first study
• You can run a smaller-scale pilot to test the practical feasibility (BUT not to test your hypotheses or the effect size!)
• Templates – there are many options out there
• We use the one from AsPredicted.org
• Submission: I recommend OSF
• Who reviews the project? - No one for content
• Sample size planning
• Data quality
• In developmental studies: exclusion criteria

Practical matters

A simple template: https://osf.io/m3spx/files/osfstorage

​

1. Have any data been collected for this study already?

2. What's the main question being asked or hypothesis being tested in this study?

3. Describe the key dependent variable(s) specifying how they will be measured.

4. How many and which conditions will participants be assigned to?

5. Specify exactly which analyses you will conduct to examine the main question/hypothesis.

Correcting for multiple comparisons

Handling missing data

Checking your assumptions

6. Any secondary analyses?

7. How many observations will be collected or what will determine the sample size? No need to justify decision, but be precise about exactly how the number will be determined.

8. Anything else you would like to pre-register? (e.g., data exclusions, variables collected for exploratory purposes, unusual analyses planned?)

Practical matters

• Analyses: how will you test your predictions

Do i need to understand the data distribution and diagnostics first before pre-registration? - Yes

Where is the sweet spot for specificity? - right now, no hard and enforced guidelines

• BUT: Preregistering ANY level of detail is better than not doing it at all!
• I suggest the level and specificity that you would use in describing your analysis in your paper
• Distinguish your main (confirmatory) and your exploratory hypotheses

​

Deviations from the plan

• You can still publish your paper! Just need to be transparent about the deviations. You need to trust your editor to weigh the severity of the issues
• Template: Willroth & Atherton, 2024, Adv Meth Prac in Psych Sci
• Forced deviations
• You said you’ll test 50 12-month-olds, +/- 1 month, but at the end, your range is +/- 2 months – I’d say it’s OK
• “What was I thinking?” deviations
• In child studies, data collection can take a looong time – you may realize you need to use some different statistical methods. If you switch before you analyze your data the original way, then you should be fine
• Totally new exploratory analyses
• This is the trickiest to evaluate theoretically

Deviations from the plan

• How much detail?

​

Other resources

• OSF (http://osf.io/prereg/) – tons of introductory materials, templates
• SREE (https://sreereg.icpsr.umich.edu/)
• Declare Design (http://declaredesign.org/) -- provides structured workflows
• Framework for Open and Reproducible Research Training (https://forrt.org)
• A consensus-based transparency checklist (Aczel et al, 2020, NHB)

​

Adopting new ways of doing things is always hard

I hope this nudge was enough to take your next step!

Thank you! Questions???

https://tinyurl.com/ManyBabiesPreregistration

Zsuzsa.Kaldy@umb.edu

babies.umb.edu